From mathew at rodley.com.au Sun Dec 1 20:28:57 2013 From: mathew at rodley.com.au (Mathew Rodley) Date: Sun, 01 Dec 2013 20:28:57 -0000 Subject: [PATCH 1 of 1] Added support for TCP_FASTOPEN supported in Linux >= 3.7.1 In-Reply-To: References: Message-ID: <39e094bd562ab0198469.1383460820@ubuntu> # HG changeset patch # User Mathew Rodley # Date 1383380887 25200 # Sat Nov 02 01:28:07 2013 -0700 # Node ID 39e094bd562ab019846906227b1fb0f8337a4ac7 # Parent dea321e5c0216efccbb23e84bbce7cf3e28f130c Added support for TCP_FASTOPEN supported in Linux >= 3.7.1 --- auto/unix | 12 ++++++++++++ src/core/ngx_connection.c | 32 ++++++++++++++++++++++++++++++++ src/core/ngx_connection.h | 4 ++++ src/http/ngx_http.c | 4 ++++ src/http/ngx_http_core_module.c | 21 +++++++++++++++++++++ src/http/ngx_http_core_module.h | 3 +++ 6 files changed, 76 insertions(+) diff -r dea321e5c021 -r 39e094bd562a auto/unix --- a/auto/unix Thu Oct 31 18:23:49 2013 +0400 +++ b/auto/unix Sat Nov 02 01:28:07 2013 -0700 @@ -344,6 +344,18 @@ . auto/feature +ngx_feature="TCP_FASTOPEN" +ngx_feature_name="NGX_HAVE_TCPFASTOPEN" +ngx_feature_run=no +ngx_feature_incs="#include + #include + #include " +ngx_feature_path= +ngx_feature_libs= +ngx_feature_test="setsockopt(0, IPPROTO_TCP, TCP_FASTOPEN, NULL, 0)" +. auto/feature + + ngx_feature="TCP_INFO" ngx_feature_name="NGX_HAVE_TCP_INFO" ngx_feature_run=no diff -r dea321e5c021 -r 39e094bd562a src/core/ngx_connection.c --- a/src/core/ngx_connection.c Thu Oct 31 18:23:49 2013 +0400 +++ b/src/core/ngx_connection.c Sat Nov 02 01:28:07 2013 -0700 @@ -82,6 +82,10 @@ ls->setfib = -1; #endif +#if (NGX_HAVE_TCPFASTOPEN) + ls->fastopen = -1; +#endif + return ls; } @@ -209,6 +213,21 @@ #endif #endif +#if (NGX_HAVE_TCPFASTOPEN) + + if (getsockopt(ls[i].fastopen, IPPROTO_TCP, TCP_FASTOPEN, + (void *) &ls[i].fastopen, &olen) + == -1) + { + ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_socket_errno, + "getsockopt(TCP_FASTOPEN) %V failed, ignored", + &ls[i].addr_text); + + ls[i].fastopen = -1; + } + +#endif + #if (NGX_HAVE_DEFERRED_ACCEPT && defined SO_ACCEPTFILTER) ngx_memzero(&af, sizeof(struct accept_filter_arg)); @@ -582,6 +601,19 @@ } #endif +#if (NGX_HAVE_TCPFASTOPEN) + if (ls[i].fastopen != -1) { + if (setsockopt(ls[i].fd, IPPROTO_TCP, TCP_FASTOPEN, + (const int*) &ls[i].fastopen, sizeof(int)) + == -1) + { + ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_socket_errno, + "setsockopt(TCP_FASTOPEN, %d) %V failed, ignored", + ls[i].fastopen, &ls[i].addr_text); + } + } +#endif + #if 0 if (1) { int tcp_nodelay = 1; diff -r dea321e5c021 -r 39e094bd562a src/core/ngx_connection.h --- a/src/core/ngx_connection.h Thu Oct 31 18:23:49 2013 +0400 +++ b/src/core/ngx_connection.h Sat Nov 02 01:28:07 2013 -0700 @@ -80,6 +80,10 @@ int setfib; #endif +#if (NGX_HAVE_TCPFASTOPEN) + int fastopen; +#endif + }; diff -r dea321e5c021 -r 39e094bd562a src/http/ngx_http.c --- a/src/http/ngx_http.c Thu Oct 31 18:23:49 2013 +0400 +++ b/src/http/ngx_http.c Sat Nov 02 01:28:07 2013 -0700 @@ -1811,6 +1811,10 @@ ls->setfib = addr->opt.setfib; #endif +#if (NGX_HAVE_TCPFASTOPEN) + ls->fastopen = addr->opt.fastopen; +#endif + return ls; } diff -r dea321e5c021 -r 39e094bd562a src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c Thu Oct 31 18:23:49 2013 +0400 +++ b/src/http/ngx_http_core_module.c Sat Nov 02 01:28:07 2013 -0700 @@ -3041,6 +3041,9 @@ #if (NGX_HAVE_SETFIB) lsopt.setfib = -1; #endif +#if (NGX_HAVE_TCPFASTOPEN) + lsopt.fastopen = -1; +#endif lsopt.wildcard = 1; (void) ngx_sock_ntop(&lsopt.u.sockaddr, lsopt.socklen, lsopt.addr, @@ -3989,6 +3992,9 @@ #if (NGX_HAVE_SETFIB) lsopt.setfib = -1; #endif +#if (NGX_HAVE_TCPFASTOPEN) + lsopt.fastopen = -1; +#endif lsopt.wildcard = u.wildcard; #if (NGX_HAVE_INET6 && defined IPV6_V6ONLY) lsopt.ipv6only = 1; @@ -4025,6 +4031,21 @@ continue; } #endif + +#if (NGX_HAVE_TCPFASTOPEN) + if (ngx_strncmp(value[n].data, "fastopen=", 9) == 0) { + lsopt.fastopen = ngx_atoi(value[n].data + 9, value[n].len - 9); + + if (lsopt.fastopen == NGX_ERROR) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "invalid fastopen \"%V\"", &value[n]); + return NGX_CONF_ERROR; + } + + continue; + } +#endif + if (ngx_strncmp(value[n].data, "backlog=", 8) == 0) { lsopt.backlog = ngx_atoi(value[n].data + 8, value[n].len - 8); lsopt.set = 1; diff -r dea321e5c021 -r 39e094bd562a src/http/ngx_http_core_module.h --- a/src/http/ngx_http_core_module.h Thu Oct 31 18:23:49 2013 +0400 +++ b/src/http/ngx_http_core_module.h Sat Nov 02 01:28:07 2013 -0700 @@ -89,6 +89,9 @@ #if (NGX_HAVE_SETFIB) int setfib; #endif +#if (NGX_HAVE_TCPFASTOPEN) + int fastopen; +#endif #if (NGX_HAVE_KEEPALIVE_TUNABLE) int tcp_keepidle; int tcp_keepintvl; From mathew at rodley.com.au Sun Dec 1 20:28:58 2013 From: mathew at rodley.com.au (Mathew Rodley) Date: Sun, 01 Dec 2013 20:28:58 -0000 Subject: [PATCH 0 of 1] Added support for TCP_FASTOPEN supported in Linux >= 3.7.1 Message-ID: Hi all, I was playing around with TCP FastOpen (an experimental TCP extension supported for clients in Linux >=3.6 and for servers in Linux >=3.7.1) that reduces the initial TCP 3-way handshake latency cost. I noticed a couple of other projects (HAProxy in particular) have support for it, and it may be useful to some people to reduce latency within their stack or experiment with it for broader deployment. It's a pretty small and non invasive change too, so following this is a patch that optionally enables it on a listener if you specify fastopen= e.g. >> listen 443 ssl fastopen=5; See: http://lwn.net/Articles/508865/ From dermiste at kilob.yt Mon Dec 2 10:22:50 2013 From: dermiste at kilob.yt (Vincent Gross) Date: Mon, 2 Dec 2013 21:22:50 +1100 Subject: Doing an "upstream" request without sending result back to client Message-ID: <20131202102250.GA28577@curry.kilob.yt> Hi folks, I want to implemement a webid authn module in nginx. This is the big picture: 1. Client do an https request and sends a client cert 2. Certs contains a subjectAltName URI, e.g. https://mysite.info/id/ 3. Server fetches https://mysite.info/id/ content 4. If content do contains public key info matching client cert, authn successful. From what I see in src/http/ngx_http_upstream.c in functions ngx_http_upstream_process_header(ngx_http_request_t *, ngx_http_upstream_t *) & ngx_http_upstream_send_response(ngx_http_request_t *, ngx_http_upstream_t *), it is not possible to do the request [3] without sending back at least a response header back to the client. Did I miss something or am I really setting myself for reimplementing the mechanisms of ngx_http_upstream.c? Any advices/guidelines for the latter (other then taking example on original upstream?) Cheers, -- Vincent -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available URL: From mdounin at mdounin.ru Mon Dec 2 13:48:40 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 2 Dec 2013 17:48:40 +0400 Subject: Doing an "upstream" request without sending result back to client In-Reply-To: <20131202102250.GA28577@curry.kilob.yt> References: <20131202102250.GA28577@curry.kilob.yt> Message-ID: <20131202134839.GV93176@mdounin.ru> Hello! On Mon, Dec 02, 2013 at 09:22:50PM +1100, Vincent Gross wrote: > Hi folks, > > I want to implemement a webid authn module in nginx. This is the big picture: > > 1. Client do an https request and sends a client cert > 2. Certs contains a subjectAltName URI, e.g. https://mysite.info/id/ > 3. Server fetches https://mysite.info/id/ content > 4. If content do contains public key info matching client cert, authn > successful. > > From what I see in src/http/ngx_http_upstream.c in functions > ngx_http_upstream_process_header(ngx_http_request_t *, ngx_http_upstream_t *) > & ngx_http_upstream_send_response(ngx_http_request_t *, ngx_http_upstream_t *), > it is not possible to do the request [3] without sending back at least a > response header back to the client. > > Did I miss something or am I really setting myself for reimplementing the > mechanisms of ngx_http_upstream.c? Any advices/guidelines for the latter (other > then taking example on original upstream?) You may try looking into subrequests, in particular - NGX_HTTP_SUBREQUEST_IN_MEMORY functionality. -- Maxim Dounin http://nginx.org/en/donation.html From dean at lightbulbone.com Mon Dec 2 23:44:53 2013 From: dean at lightbulbone.com (Dean Pucsek) Date: Mon, 2 Dec 2013 15:44:53 -0800 Subject: NGX_INT32_LEN and NGX_INT64_LEN Message-ID: Hello, While reading through the source code for nginx I came across the following two lines in ngx_config.h? #define NGX_INT32_LEN ? (sizeof("-2147483648") - 1) #define NGX_INT64_LEN ? (sizeof("-9223372036854775808") - 1) I was wondering if someone could explain the intention of these lines to me because it is not clear. ?My understanding is that using sizeof() on a string will return the number of characters in that string. ?Conversely, I get the feeling these lines are supposed to somehow act as replacements for INT32_MAX and INT64_MAX in stdint.h. Looking at code where these #define?s are used doesn?t really help clarify things either. ?For example, in nginx.c there is: ? ? var = ngx_alloc(sizeof(NGINX_VAR) ? ? ? ? ? ? ? ? ? ? + cycle->listening.nelts * (NGX_INT32_LEN + 1) + 2, ? ? ? ? ? ? ? ? ? ? cycle->log); The code clearly allocates memory, but it?s not clear why the allocation is a multiple of NGX_INT32_LEN (or why we?re adding 1, or 2 for that matter). Any direction would be much appreciated. - Dean -------------- next part -------------- An HTML attachment was scrubbed... URL: From dermiste at kilob.yt Tue Dec 3 01:26:56 2013 From: dermiste at kilob.yt (Vincent Gross) Date: Tue, 3 Dec 2013 12:26:56 +1100 Subject: Doing an "upstream" request without sending result back to client In-Reply-To: <20131202134839.GV93176@mdounin.ru> References: <20131202102250.GA28577@curry.kilob.yt> <20131202134839.GV93176@mdounin.ru> Message-ID: <20131203012656.GB30194@curry.kilob.yt> On Mon, Dec 02, 2013 at 05:48:40PM +0400, Maxim Dounin wrote: > Hello! > > On Mon, Dec 02, 2013 at 09:22:50PM +1100, Vincent Gross wrote: > > > Hi folks, > > > > I want to implemement a webid authn module in nginx. This is the big picture: > > > > 1. Client do an https request and sends a client cert > > 2. Certs contains a subjectAltName URI, e.g. https://mysite.info/id/ > > 3. Server fetches https://mysite.info/id/ content > > 4. If content do contains public key info matching client cert, authn > > successful. > > > > From what I see in src/http/ngx_http_upstream.c in functions > > ngx_http_upstream_process_header(ngx_http_request_t *, ngx_http_upstream_t *) > > & ngx_http_upstream_send_response(ngx_http_request_t *, ngx_http_upstream_t *), > > it is not possible to do the request [3] without sending back at least a > > response header back to the client. > > > > Did I miss something or am I really setting myself for reimplementing the > > mechanisms of ngx_http_upstream.c? Any advices/guidelines for the latter (other > > then taking example on original upstream?) > > You may try looking into subrequests, in particular - > NGX_HTTP_SUBREQUEST_IN_MEMORY functionality. Thank you for the reply Maxim, I gave it a quick shot based on your auth request module. As I understand it, subrequests are always done in the current server space, so if want to GET on a foreign server, I would need an other location with some substitutions to appropriately fill the proxy_pass value. Am I right ? Also this means I need to add a location whose sole purpose is to translate from inner reqs to outer reqs, I don't feel good abusing the config file in such a way, given that the whole point of webid is to have decentralized authn. Cheers, -- Vincent -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available URL: From igor at sysoev.ru Tue Dec 3 07:46:35 2013 From: igor at sysoev.ru (Igor Sysoev) Date: Tue, 3 Dec 2013 11:46:35 +0400 Subject: NGX_INT32_LEN and NGX_INT64_LEN In-Reply-To: References: Message-ID: On Dec 3, 2013, at 3:44 , Dean Pucsek wrote: > Hello, > > While reading through the source code for nginx I came across the following two lines in ngx_config.h > > #define NGX_INT32_LEN (sizeof("-2147483648") - 1) > #define NGX_INT64_LEN (sizeof("-9223372036854775808") - 1) > > I was wondering if someone could explain the intention of these lines to me because it is not clear. My understanding is that using sizeof() on a string will return the number of characters in that string. Conversely, I get the feeling these lines are supposed to somehow act as replacements for INT32_MAX and INT64_MAX in stdint.h. > > Looking at code where these #define?s are used doesn?t really help clarify things either. For example, in nginx.c there is: > > var = ngx_alloc(sizeof(NGINX_VAR) > + cycle->listening.nelts * (NGX_INT32_LEN + 1) + 2, > cycle->log); > > The code clearly allocates memory, but it?s not clear why the allocation is a multiple of NGX_INT32_LEN (or why we?re adding 1, or 2 for that matter). > > Any direction would be much appreciated. They used to allocate memory for string representation of number. In this particular case 1 is for ";", 2 is for "=" and trailing zero. -- Igor Sysoev http://nginx.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.koch007 at outlook.com Tue Dec 3 11:39:24 2013 From: alex.koch007 at outlook.com (Alex Koch) Date: Tue, 3 Dec 2013 12:39:24 +0100 Subject: NGINX Module - create variables Message-ID: Hi, I would like to create a small module which execute some routines, returns an NGX_OK, somewhat similar in concept to http://blog.zhuzhaoyuan.com/2009/08/creating-a-hello-world-nginx-module/ However I would like once the module executes to create variables such as $my_var which would be accessible via config files to other blocks or to the same location block. I am aware of the "register_variable" option, with "ngx_http_variable_t *var, *v;" - but the examples I have seen so far, execute the module only once this variable is loaded in the config file. What I would like is being able to define a couple of config variables once my module is loaded. Is this possible at all? If so, could you point me to a sample/module which does this so I can learn from it? Many thanks, Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From ru at nginx.com Tue Dec 3 12:14:32 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Tue, 03 Dec 2013 12:14:32 +0000 Subject: [nginx] Fixed null pointer dereference with $upstream_cache_last... Message-ID: details: http://hg.nginx.org/nginx/rev/b7b8e2fa7ebd branches: changeset: 5453:b7b8e2fa7ebd user: Ruslan Ermilov date: Tue Dec 03 15:11:24 2013 +0400 description: Fixed null pointer dereference with $upstream_cache_last_modified. diffstat: src/http/ngx_http_upstream.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diffs (13 lines): diff -r b7bf4671bb7b -r b7b8e2fa7ebd src/http/ngx_http_upstream.c --- a/src/http/ngx_http_upstream.c Fri Nov 29 17:23:47 2013 +0400 +++ b/src/http/ngx_http_upstream.c Tue Dec 03 15:11:24 2013 +0400 @@ -4555,7 +4555,8 @@ ngx_http_upstream_cache_last_modified(ng { u_char *p; - if (!r->upstream->conf->cache_revalidate + if (r->upstream == NULL + || !r->upstream->conf->cache_revalidate || r->upstream->cache_status != NGX_HTTP_CACHE_EXPIRED || r->cache->last_modified == -1) { From mdounin at mdounin.ru Tue Dec 3 12:38:13 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 3 Dec 2013 16:38:13 +0400 Subject: NGX_INT32_LEN and NGX_INT64_LEN In-Reply-To: References: Message-ID: <20131203123813.GE93176@mdounin.ru> Hello! On Mon, Dec 02, 2013 at 03:44:53PM -0800, Dean Pucsek wrote: > Hello, > > While reading through the source code for nginx I came across the following two lines in ngx_config.h? > > #define NGX_INT32_LEN ? (sizeof("-2147483648") - 1) > #define NGX_INT64_LEN ? (sizeof("-9223372036854775808") - 1) > > I was wondering if someone could explain the intention of these > lines to me because it is not clear. ?My understanding is that > using sizeof() on a string will return the number of characters > in that string. ?Conversely, I get the feeling these lines are > supposed to somehow act as replacements for INT32_MAX and > INT64_MAX in stdint.h. These macros are used as a maximum length of a string representation of the relevant types. Note "_LEN", not "_MAX". Relevant maximum value for int32_t is NGX_MAX_INT32_VALUE defined below in the same file. > Looking at code where these #define?s are used doesn?t really > help clarify things either. ?For example, in nginx.c there is: > > ? ? var = ngx_alloc(sizeof(NGINX_VAR) > ? ? ? ? ? ? ? ? ? ? + cycle->listening.nelts * (NGX_INT32_LEN + 1) + 2, > ? ? ? ? ? ? ? ? ? ? cycle->log); > > The code clearly allocates memory, but it?s not clear why the > allocation is a multiple of NGX_INT32_LEN (or why we?re adding > 1, or 2 for that matter). > > Any direction would be much appreciated. This code allocates memory for the "NGINX=x;y;z;\0" string, where "x", "y", "z" - are string representations of listening sockets. The "+ 1" is for ";" after each socket. The "+ 2" is for "=" and trailing "\0" (well, it looks like a only "+ 1" is actually needed here, as sizeof(NGINX_VAR) already includes an extra byte). -- Maxim Dounin http://nginx.org/en/donation.html From vl at nginx.com Tue Dec 3 13:21:49 2013 From: vl at nginx.com (Homutov Vladimir) Date: Tue, 03 Dec 2013 13:21:49 +0000 Subject: [nginx] Improved code readablity in ngx_http_upstream_init_round... Message-ID: details: http://hg.nginx.org/nginx/rev/359f49a84f87 branches: changeset: 5454:359f49a84f87 user: Vladimir Homutov date: Tue Dec 03 17:12:16 2013 +0400 description: Improved code readablity in ngx_http_upstream_init_round_robin(). Changed initialization order of the peer structure in one of the cases to be in line with the rest. No functional changes. diffstat: src/http/ngx_http_upstream_round_robin.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diffs (19 lines): diff -r b7b8e2fa7ebd -r 359f49a84f87 src/http/ngx_http_upstream_round_robin.c --- a/src/http/ngx_http_upstream_round_robin.c Tue Dec 03 15:11:24 2013 +0400 +++ b/src/http/ngx_http_upstream_round_robin.c Tue Dec 03 17:12:16 2013 +0400 @@ -79,12 +79,12 @@ ngx_http_upstream_init_round_robin(ngx_c peers->peer[n].sockaddr = server[i].addrs[j].sockaddr; peers->peer[n].socklen = server[i].addrs[j].socklen; peers->peer[n].name = server[i].addrs[j].name; + peers->peer[n].weight = server[i].weight; + peers->peer[n].effective_weight = server[i].weight; + peers->peer[n].current_weight = 0; peers->peer[n].max_fails = server[i].max_fails; peers->peer[n].fail_timeout = server[i].fail_timeout; peers->peer[n].down = server[i].down; - peers->peer[n].weight = server[i].weight; - peers->peer[n].effective_weight = server[i].weight; - peers->peer[n].current_weight = 0; n++; } } From wmark+nginx at hurrikane.de Tue Dec 3 13:49:48 2013 From: wmark+nginx at hurrikane.de (W-Mark Kubacki) Date: Tue, 3 Dec 2013 14:49:48 +0100 Subject: Nginx Logging to Zeromq Module - Sparkngin In-Reply-To: References: Message-ID: 2013/11/16 Steve Morin : > Does anyone have experience integrating zeromq with Nginx. I am looking for > some pointers, to see what concerns I should look out for. > > I am trying to contribute this code to a open source project. > -Steve This seems to me being a good template for what you want to accomplish: [1] http://www.binpress.com/app/nginx-redislog-module/998 Keep in mind that ??MQ sockets are not threadsafe? and don't share them between threads. [2] http://zguide.zeromq.org/page:all#Multithreading-with-MQ See also: [3] http://forum.nginx.org/read.php?2,243942,243986#msg-243986 (master process ? init_module callback; don't define sockets here; zmq_init goes here) [4] http://www.evanmiller.org/nginx-modules-guide.html#definition A note on performance: ? Use zero-copy wherever possible. (I would make a custom string an optional last part of a message.) ? [5] http://zeromq.org/blog:zero-copy ? Does ?MQ have UDP or similar for PUB sockets? If so, use that. ? I guess that logging or queuing through Redis will perform better than utilizing ?MQ for that matter iff everything is done on one machine under high load. -- Mark From mdounin at mdounin.ru Tue Dec 3 18:14:46 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 03 Dec 2013 18:14:46 +0000 Subject: [nginx] Fixed "setfib=" on non-first listening socket. Message-ID: details: http://hg.nginx.org/nginx/rev/8f2c69418045 branches: changeset: 5455:8f2c69418045 user: Maxim Dounin date: Tue Dec 03 21:44:08 2013 +0400 description: Fixed "setfib=" on non-first listening socket. diffstat: src/http/ngx_http_core_module.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diffs (12 lines): diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c +++ b/src/http/ngx_http_core_module.c @@ -4015,6 +4015,8 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx #if (NGX_HAVE_SETFIB) if (ngx_strncmp(value[n].data, "setfib=", 7) == 0) { lsopt.setfib = ngx_atoi(value[n].data + 7, value[n].len - 7); + lsopt.set = 1; + lsopt.bind = 1; if (lsopt.setfib == NGX_ERROR) { ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, From mdounin at mdounin.ru Tue Dec 3 18:14:47 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 03 Dec 2013 18:14:47 +0000 Subject: [nginx] Added support for TCP_FASTOPEN supported in Linux >= 3.7.1. Message-ID: details: http://hg.nginx.org/nginx/rev/692afcea9d0d branches: changeset: 5456:692afcea9d0d user: Mathew Rodley date: Tue Dec 03 22:07:03 2013 +0400 description: Added support for TCP_FASTOPEN supported in Linux >= 3.7.1. --- auto/unix | 12 ++++++++++++ src/core/ngx_connection.c | 32 ++++++++++++++++++++++++++++++++ src/core/ngx_connection.h | 4 ++++ src/http/ngx_http.c | 4 ++++ src/http/ngx_http_core_module.c | 21 +++++++++++++++++++++ src/http/ngx_http_core_module.h | 3 +++ 6 files changed, 76 insertions(+) diffstat: auto/unix | 12 ++++++++++++ src/core/ngx_connection.c | 32 ++++++++++++++++++++++++++++++++ src/core/ngx_connection.h | 4 ++++ src/http/ngx_http.c | 4 ++++ src/http/ngx_http_core_module.c | 23 +++++++++++++++++++++++ src/http/ngx_http_core_module.h | 3 +++ 6 files changed, 78 insertions(+), 0 deletions(-) diffs (166 lines): diff --git a/auto/unix b/auto/unix --- a/auto/unix +++ b/auto/unix @@ -344,6 +344,18 @@ ngx_feature_test="setsockopt(0, IPPROTO_ . auto/feature +ngx_feature="TCP_FASTOPEN" +ngx_feature_name="NGX_HAVE_TCP_FASTOPEN" +ngx_feature_run=no +ngx_feature_incs="#include + #include + #include " +ngx_feature_path= +ngx_feature_libs= +ngx_feature_test="setsockopt(0, IPPROTO_TCP, TCP_FASTOPEN, NULL, 0)" +. auto/feature + + ngx_feature="TCP_INFO" ngx_feature_name="NGX_HAVE_TCP_INFO" ngx_feature_run=no diff --git a/src/core/ngx_connection.c b/src/core/ngx_connection.c --- a/src/core/ngx_connection.c +++ b/src/core/ngx_connection.c @@ -82,6 +82,10 @@ ngx_create_listening(ngx_conf_t *cf, voi ls->setfib = -1; #endif +#if (NGX_HAVE_TCP_FASTOPEN) + ls->fastopen = -1; +#endif + return ls; } @@ -209,6 +213,21 @@ ngx_set_inherited_sockets(ngx_cycle_t *c #endif #endif +#if (NGX_HAVE_TCP_FASTOPEN) + + if (getsockopt(ls[i].fastopen, IPPROTO_TCP, TCP_FASTOPEN, + (void *) &ls[i].fastopen, &olen) + == -1) + { + ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_socket_errno, + "getsockopt(TCP_FASTOPEN) %V failed, ignored", + &ls[i].addr_text); + + ls[i].fastopen = -1; + } + +#endif + #if (NGX_HAVE_DEFERRED_ACCEPT && defined SO_ACCEPTFILTER) ngx_memzero(&af, sizeof(struct accept_filter_arg)); @@ -582,6 +601,19 @@ ngx_configure_listening_sockets(ngx_cycl } #endif +#if (NGX_HAVE_TCP_FASTOPEN) + if (ls[i].fastopen != -1) { + if (setsockopt(ls[i].fd, IPPROTO_TCP, TCP_FASTOPEN, + (const void *) &ls[i].fastopen, sizeof(int)) + == -1) + { + ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_socket_errno, + "setsockopt(TCP_FASTOPEN, %d) %V failed, ignored", + ls[i].fastopen, &ls[i].addr_text); + } + } +#endif + #if 0 if (1) { int tcp_nodelay = 1; diff --git a/src/core/ngx_connection.h b/src/core/ngx_connection.h --- a/src/core/ngx_connection.h +++ b/src/core/ngx_connection.h @@ -80,6 +80,10 @@ struct ngx_listening_s { int setfib; #endif +#if (NGX_HAVE_TCP_FASTOPEN) + int fastopen; +#endif + }; diff --git a/src/http/ngx_http.c b/src/http/ngx_http.c --- a/src/http/ngx_http.c +++ b/src/http/ngx_http.c @@ -1811,6 +1811,10 @@ ngx_http_add_listening(ngx_conf_t *cf, n ls->setfib = addr->opt.setfib; #endif +#if (NGX_HAVE_TCP_FASTOPEN) + ls->fastopen = addr->opt.fastopen; +#endif + return ls; } diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c +++ b/src/http/ngx_http_core_module.c @@ -3041,6 +3041,9 @@ ngx_http_core_server(ngx_conf_t *cf, ngx #if (NGX_HAVE_SETFIB) lsopt.setfib = -1; #endif +#if (NGX_HAVE_TCP_FASTOPEN) + lsopt.fastopen = -1; +#endif lsopt.wildcard = 1; (void) ngx_sock_ntop(&lsopt.u.sockaddr, lsopt.socklen, lsopt.addr, @@ -3989,6 +3992,9 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx #if (NGX_HAVE_SETFIB) lsopt.setfib = -1; #endif +#if (NGX_HAVE_TCP_FASTOPEN) + lsopt.fastopen = -1; +#endif lsopt.wildcard = u.wildcard; #if (NGX_HAVE_INET6 && defined IPV6_V6ONLY) lsopt.ipv6only = 1; @@ -4027,6 +4033,23 @@ ngx_http_core_listen(ngx_conf_t *cf, ngx continue; } #endif + +#if (NGX_HAVE_TCP_FASTOPEN) + if (ngx_strncmp(value[n].data, "fastopen=", 9) == 0) { + lsopt.fastopen = ngx_atoi(value[n].data + 9, value[n].len - 9); + lsopt.set = 1; + lsopt.bind = 1; + + if (lsopt.fastopen == NGX_ERROR) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "invalid fastopen \"%V\"", &value[n]); + return NGX_CONF_ERROR; + } + + continue; + } +#endif + if (ngx_strncmp(value[n].data, "backlog=", 8) == 0) { lsopt.backlog = ngx_atoi(value[n].data + 8, value[n].len - 8); lsopt.set = 1; diff --git a/src/http/ngx_http_core_module.h b/src/http/ngx_http_core_module.h --- a/src/http/ngx_http_core_module.h +++ b/src/http/ngx_http_core_module.h @@ -89,6 +89,9 @@ typedef struct { #if (NGX_HAVE_SETFIB) int setfib; #endif +#if (NGX_HAVE_TCP_FASTOPEN) + int fastopen; +#endif #if (NGX_HAVE_KEEPALIVE_TUNABLE) int tcp_keepidle; int tcp_keepintvl; From mdounin at mdounin.ru Tue Dec 3 18:16:38 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 3 Dec 2013 22:16:38 +0400 Subject: [PATCH 1 of 1] Added support for TCP_FASTOPEN supported in Linux >= 3.7.1 In-Reply-To: <39e094bd562ab0198469.1383460820@ubuntu> References: <39e094bd562ab0198469.1383460820@ubuntu> Message-ID: <20131203181637.GM93176@mdounin.ru> Hello! On Sat, Nov 02, 2013 at 11:40:20PM -0700, Mathew Rodley wrote: > # HG changeset patch > # User Mathew Rodley > # Date 1383380887 25200 > # Sat Nov 02 01:28:07 2013 -0700 Just a side note: it looks like this patch was sitting in your mail queue about a month... > # Node ID 39e094bd562ab019846906227b1fb0f8337a4ac7 > # Parent dea321e5c0216efccbb23e84bbce7cf3e28f130c > Added support for TCP_FASTOPEN supported in Linux >= 3.7.1 Committed with minor changes (notably, NGX_HAVE_TCPFASTOPEN -> NGX_HAVE_TCP_FASTOPEN, lsopt.set/bind added). Thank you. [...] -- Maxim Dounin http://nginx.org/en/donation.html From dean at lightbulbone.com Wed Dec 4 00:43:45 2013 From: dean at lightbulbone.com (Dean Pucsek) Date: Tue, 3 Dec 2013 16:43:45 -0800 Subject: NGX_INT32_LEN and NGX_INT64_LEN In-Reply-To: <20131203123813.GE93176@mdounin.ru> References: <20131203123813.GE93176@mdounin.ru> Message-ID: On December 3, 2013 at 4:38:18 AM, Maxim Dounin (mdounin at mdounin.ru) wrote: Hello! On Mon, Dec 02, 2013 at 03:44:53PM -0800, Dean Pucsek wrote: > Hello, >? > While reading through the source code for nginx I came across the following two lines in ngx_config.h? >? > #define NGX_INT32_LEN ? (sizeof("-2147483648") - 1) > #define NGX_INT64_LEN ? (sizeof("-9223372036854775808") - 1) >? > I was wondering if someone could explain the intention of these? > lines to me because it is not clear. ?My understanding is that? > using sizeof() on a string will return the number of characters? > in that string. ?Conversely, I get the feeling these lines are? > supposed to somehow act as replacements for INT32_MAX and? > INT64_MAX in stdint.h. These macros are used as a maximum length of a string? representation of the relevant types. Note "_LEN", not "_MAX". Relevant maximum value for int32_t is NGX_MAX_INT32_VALUE defined? below in the same file. Noted, sounds like this may be a case of not fully reading ngx_config.h on my part. > Looking at code where these #define?s are used doesn?t really? > help clarify things either. ?For example, in nginx.c there is: >? > ? ? var = ngx_alloc(sizeof(NGINX_VAR) > ? ? ? ? ? ? ? ? ? ? + cycle->listening.nelts * (NGX_INT32_LEN + 1) + 2, > ? ? ? ? ? ? ? ? ? ? cycle->log); >? > The code clearly allocates memory, but it?s not clear why the? > allocation is a multiple of NGX_INT32_LEN (or why we?re adding? > 1, or 2 for that matter). >? > Any direction would be much appreciated. This code allocates memory for the "NGINX=x;y;z;\0" string, where? "x", "y", "z" - are string representations of listening sockets.? The "+ 1" is for ";" after each socket. The "+ 2" is for "="? and trailing "\0" (well, it looks like a only "+ 1" is actually? needed here, as sizeof(NGINX_VAR) already includes an extra byte). I had a suspicion that it was something like that but I didn?t quite get the pieces together. ? Thanks for the clarification. --? Maxim Dounin http://nginx.org/en/donation.html _______________________________________________ nginx-devel mailing list nginx-devel at nginx.org http://mailman.nginx.org/mailman/listinfo/nginx-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Wed Dec 4 15:51:19 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 4 Dec 2013 19:51:19 +0400 Subject: [PATCH v2] uwsgi over ssl In-Reply-To: <0dc9a0b4c852ede3d20403e82a1d50fe.squirrel@manage.unbit.it> References: <0dc9a0b4c852ede3d20403e82a1d50fe.squirrel@manage.unbit.it> Message-ID: <20131204155119.GS93176@mdounin.ru> Hello! On Tue, Nov 19, 2013 at 11:24:50AM +0100, Roberto De Ioris wrote: > Hi, this is a new patch for uwsgi over ssl support aimed at nginx 1.5.x > > It now exposes 4 options: > > uwsgi_ssl > > uwsgi_ssl_session_reuse > > uwsgi_ssl_protocols > > uwsgi_ssl_ciphers Sorry for long delay. I've looked into this, and I tend to think that "uwsgi_ssl" is a wrong aproach. E.g., consider the following configuration: location / { uwsgi_pass upstream1; uwsgi_ssl on; location /nested/ { uwsgi_pass upstream2; uwsgi_ssl_protocols TLSv2; } } Requests to upstream2 will use SSL, but uwsgi_ssl_protocols won't have any effect. While this is easy to fix, this is certainly counter-intuitive. Instead, I think it would be better to use something like this: uwsgi_pass uwsgis://upstream; Where "uwsgis" scheme means "uwsgi over ssl", much like in https with proxy_pass. This way, SSL is configured with the uwsgi_pass and don't have unexpected side effects like the above. Below is a patch which implements this aproach. What do you think? # HG changeset patch # User Maxim Dounin # Date 1386171311 -14400 # Wed Dec 04 19:35:11 2013 +0400 # Node ID 289c2c5ccc381cea89b69cac0cb130b0b9b4494c # Parent 692afcea9d0d05c10b1dd7f2bd095dd014665f4c SSL support in the uwsgi module. Based on patch by Roberto De Ioris. diff --git a/src/http/modules/ngx_http_uwsgi_module.c b/src/http/modules/ngx_http_uwsgi_module.c --- a/src/http/modules/ngx_http_uwsgi_module.c +++ b/src/http/modules/ngx_http_uwsgi_module.c @@ -34,6 +34,12 @@ typedef struct { ngx_uint_t modifier1; ngx_uint_t modifier2; + +#if (NGX_HTTP_SSL) + ngx_uint_t ssl; + ngx_uint_t ssl_protocols; + ngx_str_t ssl_ciphers; +#endif } ngx_http_uwsgi_loc_conf_t; @@ -66,6 +72,11 @@ static char *ngx_http_uwsgi_cache_key(ng void *conf); #endif +#if (NGX_HTTP_SSL) +static ngx_int_t ngx_http_uwsgi_set_ssl(ngx_conf_t *cf, + ngx_http_uwsgi_loc_conf_t *uwcf); +#endif + static ngx_conf_num_bounds_t ngx_http_uwsgi_modifier_bounds = { ngx_conf_check_num_bounds, 0, 255 @@ -86,6 +97,20 @@ static ngx_conf_bitmask_t ngx_http_uwsgi }; +#if (NGX_HTTP_SSL) + +static ngx_conf_bitmask_t ngx_http_uwsgi_ssl_protocols[] = { + { ngx_string("SSLv2"), NGX_SSL_SSLv2 }, + { ngx_string("SSLv3"), NGX_SSL_SSLv3 }, + { ngx_string("TLSv1"), NGX_SSL_TLSv1 }, + { ngx_string("TLSv1.1"), NGX_SSL_TLSv1_1 }, + { ngx_string("TLSv1.2"), NGX_SSL_TLSv1_2 }, + { ngx_null_string, 0 } +}; + +#endif + + ngx_module_t ngx_http_uwsgi_module; @@ -361,6 +386,31 @@ static ngx_command_t ngx_http_uwsgi_comm offsetof(ngx_http_uwsgi_loc_conf_t, upstream.ignore_headers), &ngx_http_upstream_ignore_headers_masks }, +#if (NGX_HTTP_SSL) + + { ngx_string("uwsgi_ssl_session_reuse"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_uwsgi_loc_conf_t, upstream.ssl_session_reuse), + NULL }, + + { ngx_string("uwsgi_ssl_protocols"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_1MORE, + ngx_conf_set_bitmask_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_uwsgi_loc_conf_t, ssl_protocols), + &ngx_http_uwsgi_ssl_protocols }, + + { ngx_string("uwsgi_ssl_ciphers"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, + ngx_conf_set_str_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_uwsgi_loc_conf_t, ssl_ciphers), + NULL }, + +#endif + ngx_null_command }; @@ -448,15 +498,29 @@ ngx_http_uwsgi_handler(ngx_http_request_ uwcf = ngx_http_get_module_loc_conf(r, ngx_http_uwsgi_module); - if (uwcf->uwsgi_lengths) { + u = r->upstream; + + if (uwcf->uwsgi_lengths == NULL) { + +#if (NGX_HTTP_SSL) + u->ssl = (uwcf->upstream.ssl != NULL); + + if (u->ssl) { + ngx_str_set(&u->schema, "uwsgis://"); + + } else { + ngx_str_set(&u->schema, "uwsgi://"); + } +#else + ngx_str_set(&u->schema, "uwsgi://"); +#endif + + } else { if (ngx_http_uwsgi_eval(r, uwcf) != NGX_OK) { return NGX_HTTP_INTERNAL_SERVER_ERROR; } } - u = r->upstream; - - ngx_str_set(&u->schema, "uwsgi://"); u->output.tag = (ngx_buf_tag_t) &ngx_http_uwsgi_module; u->conf = &uwcf->upstream; @@ -494,6 +558,7 @@ ngx_http_uwsgi_handler(ngx_http_request_ static ngx_int_t ngx_http_uwsgi_eval(ngx_http_request_t *r, ngx_http_uwsgi_loc_conf_t * uwcf) { + size_t add; ngx_url_t url; ngx_http_upstream_t *u; @@ -506,6 +571,41 @@ ngx_http_uwsgi_eval(ngx_http_request_t * return NGX_ERROR; } + if (url.url.len > 8 + && ngx_strncasecmp(url.url.data, (u_char *) "uwsgi://", 8) == 0) + { + add = 8; + + } else if (url.url.len > 9 + && ngx_strncasecmp(url.url.data, (u_char *) "uwsgis://", 9) == 0) + { + +#if (NGX_HTTP_SSL) + add = 9; + r->upstream->ssl = 1; +#else + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "uwsgis protocol requires SSL support"); + return NGX_CONF_ERROR; +#endif + + } else { + add = 0; + } + + u = r->upstream; + + if (add) { + u->schema.len = add; + u->schema.data = url.url.data; + + url.url.data += add; + url.url.len -= add; + + } else { + ngx_str_set(&u->schema, (u_char *) "uwsgi://"); + } + url.no_resolve = 1; if (ngx_parse_url(r->pool, &url) != NGX_OK) { @@ -517,8 +617,6 @@ ngx_http_uwsgi_eval(ngx_http_request_t * return NGX_ERROR; } - u = r->upstream; - u->resolved = ngx_pcalloc(r->pool, sizeof(ngx_http_upstream_resolved_t)); if (u->resolved == NULL) { return NGX_ERROR; @@ -1145,6 +1243,9 @@ ngx_http_uwsgi_create_loc_conf(ngx_conf_ conf->upstream.pass_headers = NGX_CONF_UNSET_PTR; conf->upstream.intercept_errors = NGX_CONF_UNSET; +#if (NGX_HTTP_SSL) + conf->upstream.ssl_session_reuse = NGX_CONF_UNSET; +#endif /* "uwsgi_cyclic_temp_file" is disabled */ conf->upstream.cyclic_temp_file = 0; @@ -1392,6 +1493,27 @@ ngx_http_uwsgi_merge_loc_conf(ngx_conf_t ngx_conf_merge_value(conf->upstream.intercept_errors, prev->upstream.intercept_errors, 0); +#if (NGX_HTTP_SSL) + ngx_conf_merge_value(conf->upstream.ssl_session_reuse, + prev->upstream.ssl_session_reuse, 1); + + ngx_conf_merge_bitmask_value(conf->ssl_protocols, prev->ssl_protocols, + (NGX_CONF_BITMASK_SET|NGX_SSL_SSLv3 + |NGX_SSL_TLSv1|NGX_SSL_TLSv1_1 + |NGX_SSL_TLSv1_2)); + + ngx_conf_merge_str_value(conf->ssl_ciphers, prev->ssl_ciphers, + "DEFAULT"); + + if (conf->ssl && ngx_http_uwsgi_set_ssl(cf, conf) != NGX_OK) { + return NGX_CONF_ERROR; + } + + if (conf->upstream.ssl == NULL) { + conf->upstream.ssl = prev->upstream.ssl; + } +#endif + ngx_conf_merge_str_value(conf->uwsgi_string, prev->uwsgi_string, ""); hash.max_size = 512; @@ -1670,6 +1792,7 @@ ngx_http_uwsgi_pass(ngx_conf_t *cf, ngx_ { ngx_http_uwsgi_loc_conf_t *uwcf = conf; + size_t add; ngx_url_t u; ngx_str_t *value, *url; ngx_uint_t n; @@ -1705,12 +1828,35 @@ ngx_http_uwsgi_pass(ngx_conf_t *cf, ngx_ return NGX_CONF_ERROR; } +#if (NGX_HTTP_SSL) + uwcf->ssl = 1; +#endif + return NGX_CONF_OK; } + if (ngx_strncasecmp(url->data, (u_char *) "uwsgi://", 8) == 0) { + add = 8; + + } else if (ngx_strncasecmp(url->data, (u_char *) "uwsgis://", 9) == 0) { + +#if (NGX_HTTP_SSL) + add = 9; + uwcf->ssl = 1; +#else + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "uwsgis protocol requires SSL support"); + return NGX_CONF_ERROR; +#endif + + } else { + add = 0; + } + ngx_memzero(&u, sizeof(ngx_url_t)); - u.url = value[1]; + u.url.len = url->len - add; + u.url.data = url->data + add; u.no_resolve = 1; uwcf->upstream.upstream = ngx_http_upstream_add(cf, &u, 0); @@ -1844,3 +1990,47 @@ ngx_http_uwsgi_cache_key(ngx_conf_t *cf, } #endif + + +#if (NGX_HTTP_SSL) + +static ngx_int_t +ngx_http_uwsgi_set_ssl(ngx_conf_t *cf, ngx_http_uwsgi_loc_conf_t *uwcf) +{ + ngx_pool_cleanup_t *cln; + + uwcf->upstream.ssl = ngx_pcalloc(cf->pool, sizeof(ngx_ssl_t)); + if (uwcf->upstream.ssl == NULL) { + return NGX_ERROR; + } + + uwcf->upstream.ssl->log = cf->log; + + if (ngx_ssl_create(uwcf->upstream.ssl, uwcf->ssl_protocols, NULL) + != NGX_OK) + { + return NGX_ERROR; + } + + if (SSL_CTX_set_cipher_list(uwcf->upstream.ssl->ctx, + (const char *) uwcf->ssl_ciphers.data) + == 0) + { + ngx_ssl_error(NGX_LOG_EMERG, cf->log, 0, + "SSL_CTX_set_cipher_list(\"%V\") failed", + &uwcf->ssl_ciphers); + return NGX_ERROR; + } + + cln = ngx_pool_cleanup_add(cf->pool, 0); + if (cln == NULL) { + return NGX_ERROR; + } + + cln->handler = ngx_ssl_cleanup_ctx; + cln->data = uwcf->upstream.ssl; + + return NGX_OK; +} + +#endif -- Maxim Dounin http://nginx.org/en/donation.html From roberto at unbit.it Wed Dec 4 15:55:53 2013 From: roberto at unbit.it (Roberto De Ioris) Date: Wed, 4 Dec 2013 16:55:53 +0100 Subject: [PATCH v2] uwsgi over ssl In-Reply-To: <20131204155119.GS93176@mdounin.ru> References: <0dc9a0b4c852ede3d20403e82a1d50fe.squirrel@manage.unbit.it> <20131204155119.GS93176@mdounin.ru> Message-ID: > Hello! > > On Tue, Nov 19, 2013 at 11:24:50AM +0100, Roberto De Ioris wrote: > >> Hi, this is a new patch for uwsgi over ssl support aimed at nginx 1.5.x >> >> It now exposes 4 options: >> >> uwsgi_ssl >> >> uwsgi_ssl_session_reuse >> >> uwsgi_ssl_protocols >> >> uwsgi_ssl_ciphers > > Sorry for long delay. I've looked into this, and I tend to think > that "uwsgi_ssl" is a wrong aproach. E.g., consider the following > configuration: > > location / { > uwsgi_pass upstream1; > uwsgi_ssl on; > > location /nested/ { > uwsgi_pass upstream2; > uwsgi_ssl_protocols TLSv2; > } > } > > Requests to upstream2 will use SSL, but uwsgi_ssl_protocols won't > have any effect. While this is easy to fix, this is certainly > counter-intuitive. > > Instead, I think it would be better to use something like this: > > uwsgi_pass uwsgis://upstream; > in uWSGI it is marked as "suwsgi" (secure uwsgi), so i think it would be better to call it in the same way ;) A part from this it is +1 for me -- Roberto De Ioris http://unbit.it From mdounin at mdounin.ru Wed Dec 4 19:02:24 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 04 Dec 2013 19:02:24 +0000 Subject: [nginx] SSL support in the uwsgi module. Message-ID: details: http://hg.nginx.org/nginx/rev/c82b2e020b9f branches: changeset: 5457:c82b2e020b9f user: Maxim Dounin date: Wed Dec 04 23:01:27 2013 +0400 description: SSL support in the uwsgi module. Based on patch by Roberto De Ioris. diffstat: src/http/modules/ngx_http_uwsgi_module.c | 204 +++++++++++++++++++++++++++++- 1 files changed, 197 insertions(+), 7 deletions(-) diffs (truncated from 305 to 300 lines): diff --git a/src/http/modules/ngx_http_uwsgi_module.c b/src/http/modules/ngx_http_uwsgi_module.c --- a/src/http/modules/ngx_http_uwsgi_module.c +++ b/src/http/modules/ngx_http_uwsgi_module.c @@ -34,6 +34,12 @@ typedef struct { ngx_uint_t modifier1; ngx_uint_t modifier2; + +#if (NGX_HTTP_SSL) + ngx_uint_t ssl; + ngx_uint_t ssl_protocols; + ngx_str_t ssl_ciphers; +#endif } ngx_http_uwsgi_loc_conf_t; @@ -66,6 +72,11 @@ static char *ngx_http_uwsgi_cache_key(ng void *conf); #endif +#if (NGX_HTTP_SSL) +static ngx_int_t ngx_http_uwsgi_set_ssl(ngx_conf_t *cf, + ngx_http_uwsgi_loc_conf_t *uwcf); +#endif + static ngx_conf_num_bounds_t ngx_http_uwsgi_modifier_bounds = { ngx_conf_check_num_bounds, 0, 255 @@ -86,6 +97,20 @@ static ngx_conf_bitmask_t ngx_http_uwsgi }; +#if (NGX_HTTP_SSL) + +static ngx_conf_bitmask_t ngx_http_uwsgi_ssl_protocols[] = { + { ngx_string("SSLv2"), NGX_SSL_SSLv2 }, + { ngx_string("SSLv3"), NGX_SSL_SSLv3 }, + { ngx_string("TLSv1"), NGX_SSL_TLSv1 }, + { ngx_string("TLSv1.1"), NGX_SSL_TLSv1_1 }, + { ngx_string("TLSv1.2"), NGX_SSL_TLSv1_2 }, + { ngx_null_string, 0 } +}; + +#endif + + ngx_module_t ngx_http_uwsgi_module; @@ -361,6 +386,31 @@ static ngx_command_t ngx_http_uwsgi_comm offsetof(ngx_http_uwsgi_loc_conf_t, upstream.ignore_headers), &ngx_http_upstream_ignore_headers_masks }, +#if (NGX_HTTP_SSL) + + { ngx_string("uwsgi_ssl_session_reuse"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_uwsgi_loc_conf_t, upstream.ssl_session_reuse), + NULL }, + + { ngx_string("uwsgi_ssl_protocols"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_1MORE, + ngx_conf_set_bitmask_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_uwsgi_loc_conf_t, ssl_protocols), + &ngx_http_uwsgi_ssl_protocols }, + + { ngx_string("uwsgi_ssl_ciphers"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, + ngx_conf_set_str_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_uwsgi_loc_conf_t, ssl_ciphers), + NULL }, + +#endif + ngx_null_command }; @@ -448,15 +498,29 @@ ngx_http_uwsgi_handler(ngx_http_request_ uwcf = ngx_http_get_module_loc_conf(r, ngx_http_uwsgi_module); - if (uwcf->uwsgi_lengths) { + u = r->upstream; + + if (uwcf->uwsgi_lengths == NULL) { + +#if (NGX_HTTP_SSL) + u->ssl = (uwcf->upstream.ssl != NULL); + + if (u->ssl) { + ngx_str_set(&u->schema, "suwsgi://"); + + } else { + ngx_str_set(&u->schema, "uwsgi://"); + } +#else + ngx_str_set(&u->schema, "uwsgi://"); +#endif + + } else { if (ngx_http_uwsgi_eval(r, uwcf) != NGX_OK) { return NGX_HTTP_INTERNAL_SERVER_ERROR; } } - u = r->upstream; - - ngx_str_set(&u->schema, "uwsgi://"); u->output.tag = (ngx_buf_tag_t) &ngx_http_uwsgi_module; u->conf = &uwcf->upstream; @@ -494,6 +558,7 @@ ngx_http_uwsgi_handler(ngx_http_request_ static ngx_int_t ngx_http_uwsgi_eval(ngx_http_request_t *r, ngx_http_uwsgi_loc_conf_t * uwcf) { + size_t add; ngx_url_t url; ngx_http_upstream_t *u; @@ -506,6 +571,41 @@ ngx_http_uwsgi_eval(ngx_http_request_t * return NGX_ERROR; } + if (url.url.len > 8 + && ngx_strncasecmp(url.url.data, (u_char *) "uwsgi://", 8) == 0) + { + add = 8; + + } else if (url.url.len > 9 + && ngx_strncasecmp(url.url.data, (u_char *) "suwsgi://", 9) == 0) + { + +#if (NGX_HTTP_SSL) + add = 9; + r->upstream->ssl = 1; +#else + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "suwsgi protocol requires SSL support"); + return NGX_CONF_ERROR; +#endif + + } else { + add = 0; + } + + u = r->upstream; + + if (add) { + u->schema.len = add; + u->schema.data = url.url.data; + + url.url.data += add; + url.url.len -= add; + + } else { + ngx_str_set(&u->schema, (u_char *) "uwsgi://"); + } + url.no_resolve = 1; if (ngx_parse_url(r->pool, &url) != NGX_OK) { @@ -517,8 +617,6 @@ ngx_http_uwsgi_eval(ngx_http_request_t * return NGX_ERROR; } - u = r->upstream; - u->resolved = ngx_pcalloc(r->pool, sizeof(ngx_http_upstream_resolved_t)); if (u->resolved == NULL) { return NGX_ERROR; @@ -1145,6 +1243,9 @@ ngx_http_uwsgi_create_loc_conf(ngx_conf_ conf->upstream.pass_headers = NGX_CONF_UNSET_PTR; conf->upstream.intercept_errors = NGX_CONF_UNSET; +#if (NGX_HTTP_SSL) + conf->upstream.ssl_session_reuse = NGX_CONF_UNSET; +#endif /* "uwsgi_cyclic_temp_file" is disabled */ conf->upstream.cyclic_temp_file = 0; @@ -1392,6 +1493,27 @@ ngx_http_uwsgi_merge_loc_conf(ngx_conf_t ngx_conf_merge_value(conf->upstream.intercept_errors, prev->upstream.intercept_errors, 0); +#if (NGX_HTTP_SSL) + ngx_conf_merge_value(conf->upstream.ssl_session_reuse, + prev->upstream.ssl_session_reuse, 1); + + ngx_conf_merge_bitmask_value(conf->ssl_protocols, prev->ssl_protocols, + (NGX_CONF_BITMASK_SET|NGX_SSL_SSLv3 + |NGX_SSL_TLSv1|NGX_SSL_TLSv1_1 + |NGX_SSL_TLSv1_2)); + + ngx_conf_merge_str_value(conf->ssl_ciphers, prev->ssl_ciphers, + "DEFAULT"); + + if (conf->ssl && ngx_http_uwsgi_set_ssl(cf, conf) != NGX_OK) { + return NGX_CONF_ERROR; + } + + if (conf->upstream.ssl == NULL) { + conf->upstream.ssl = prev->upstream.ssl; + } +#endif + ngx_conf_merge_str_value(conf->uwsgi_string, prev->uwsgi_string, ""); hash.max_size = 512; @@ -1670,6 +1792,7 @@ ngx_http_uwsgi_pass(ngx_conf_t *cf, ngx_ { ngx_http_uwsgi_loc_conf_t *uwcf = conf; + size_t add; ngx_url_t u; ngx_str_t *value, *url; ngx_uint_t n; @@ -1705,12 +1828,35 @@ ngx_http_uwsgi_pass(ngx_conf_t *cf, ngx_ return NGX_CONF_ERROR; } +#if (NGX_HTTP_SSL) + uwcf->ssl = 1; +#endif + return NGX_CONF_OK; } + if (ngx_strncasecmp(url->data, (u_char *) "uwsgi://", 8) == 0) { + add = 8; + + } else if (ngx_strncasecmp(url->data, (u_char *) "suwsgi://", 9) == 0) { + +#if (NGX_HTTP_SSL) + add = 9; + uwcf->ssl = 1; +#else + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "suwsgi protocol requires SSL support"); + return NGX_CONF_ERROR; +#endif + + } else { + add = 0; + } + ngx_memzero(&u, sizeof(ngx_url_t)); - u.url = value[1]; + u.url.len = url->len - add; + u.url.data = url->data + add; u.no_resolve = 1; uwcf->upstream.upstream = ngx_http_upstream_add(cf, &u, 0); @@ -1844,3 +1990,47 @@ ngx_http_uwsgi_cache_key(ngx_conf_t *cf, } #endif + + +#if (NGX_HTTP_SSL) + +static ngx_int_t +ngx_http_uwsgi_set_ssl(ngx_conf_t *cf, ngx_http_uwsgi_loc_conf_t *uwcf) +{ + ngx_pool_cleanup_t *cln; + + uwcf->upstream.ssl = ngx_pcalloc(cf->pool, sizeof(ngx_ssl_t)); + if (uwcf->upstream.ssl == NULL) { + return NGX_ERROR; + } + + uwcf->upstream.ssl->log = cf->log; + + if (ngx_ssl_create(uwcf->upstream.ssl, uwcf->ssl_protocols, NULL) + != NGX_OK) + { + return NGX_ERROR; + } + + if (SSL_CTX_set_cipher_list(uwcf->upstream.ssl->ctx, + (const char *) uwcf->ssl_ciphers.data) + == 0) + { + ngx_ssl_error(NGX_LOG_EMERG, cf->log, 0, + "SSL_CTX_set_cipher_list(\"%V\") failed", + &uwcf->ssl_ciphers); + return NGX_ERROR; + } + + cln = ngx_pool_cleanup_add(cf->pool, 0); + if (cln == NULL) { + return NGX_ERROR; + } + + cln->handler = ngx_ssl_cleanup_ctx; + cln->data = uwcf->upstream.ssl; From mdounin at mdounin.ru Wed Dec 4 19:02:52 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 4 Dec 2013 23:02:52 +0400 Subject: [PATCH v2] uwsgi over ssl In-Reply-To: References: <0dc9a0b4c852ede3d20403e82a1d50fe.squirrel@manage.unbit.it> <20131204155119.GS93176@mdounin.ru> Message-ID: <20131204190252.GT93176@mdounin.ru> Hello! On Wed, Dec 04, 2013 at 04:55:53PM +0100, Roberto De Ioris wrote: > > > Hello! > > > > On Tue, Nov 19, 2013 at 11:24:50AM +0100, Roberto De Ioris wrote: > > > >> Hi, this is a new patch for uwsgi over ssl support aimed at nginx 1.5.x > >> > >> It now exposes 4 options: > >> > >> uwsgi_ssl > >> > >> uwsgi_ssl_session_reuse > >> > >> uwsgi_ssl_protocols > >> > >> uwsgi_ssl_ciphers > > > > Sorry for long delay. I've looked into this, and I tend to think > > that "uwsgi_ssl" is a wrong aproach. E.g., consider the following > > configuration: > > > > location / { > > uwsgi_pass upstream1; > > uwsgi_ssl on; > > > > location /nested/ { > > uwsgi_pass upstream2; > > uwsgi_ssl_protocols TLSv2; > > } > > } > > > > Requests to upstream2 will use SSL, but uwsgi_ssl_protocols won't > > have any effect. While this is easy to fix, this is certainly > > counter-intuitive. > > > > Instead, I think it would be better to use something like this: > > > > uwsgi_pass uwsgis://upstream; > > > > in uWSGI it is marked as "suwsgi" (secure uwsgi), so i think it would be > better to call it in the same way ;) > > A part from this it is +1 for me Renamed and committed, thanks! -- Maxim Dounin http://nginx.org/en/donation.html From mdounin at mdounin.ru Wed Dec 4 20:20:29 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 04 Dec 2013 20:20:29 +0000 Subject: [nginx] Fixed build without SSL, broken by c82b2e020b9f. Message-ID: details: http://hg.nginx.org/nginx/rev/4a71ef1aa36d branches: changeset: 5458:4a71ef1aa36d user: Maxim Dounin date: Wed Dec 04 23:13:13 2013 +0400 description: Fixed build without SSL, broken by c82b2e020b9f. diffstat: src/http/modules/ngx_http_uwsgi_module.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diffs (16 lines): diff --git a/src/http/modules/ngx_http_uwsgi_module.c b/src/http/modules/ngx_http_uwsgi_module.c --- a/src/http/modules/ngx_http_uwsgi_module.c +++ b/src/http/modules/ngx_http_uwsgi_module.c @@ -584,9 +584,9 @@ ngx_http_uwsgi_eval(ngx_http_request_t * add = 9; r->upstream->ssl = 1; #else - ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, - "suwsgi protocol requires SSL support"); - return NGX_CONF_ERROR; + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, + "suwsgi protocol requires SSL support"); + return NGX_ERROR; #endif } else { From reeteshr at outlook.com Wed Dec 4 21:09:28 2013 From: reeteshr at outlook.com (Reetesh Ranjan) Date: Thu, 5 Dec 2013 02:39:28 +0530 Subject: How to implement handshake in an upstream module? Message-ID: Before I send my main request and process the response through create_request and process_header (and filter) callbacks, I need to have a short handshake with the upstream servers. It consists of a send() and a recv() from the upstream module. How to implement this? Would the following sequence work?States: H = do-handshake (initial state), R = do-request1 In state H, send handshake req first through create_request()2 In process_header() in state == H 2.1 call 'create_request' again with state set to R, so main request gets created 2.2 call 'ngx_http_upstream_send_request' manually to restart the req-response cycle3 Because of 2.2 we get a process_header() call in state R Regards,Reetesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Wed Dec 4 22:38:35 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 5 Dec 2013 02:38:35 +0400 Subject: How to implement handshake in an upstream module? In-Reply-To: References: Message-ID: <20131204223835.GZ93176@mdounin.ru> Hello! On Thu, Dec 05, 2013 at 02:39:28AM +0530, Reetesh Ranjan wrote: > Before I send my main request and process the response through > create_request and process_header (and filter) callbacks, I need > to have a short handshake with the upstream servers. It consists > of a send() and a recv() from the upstream module. How to > implement this? > Would the following sequence work?States: H = do-handshake > (initial state), R = do-request1 In state H, send handshake > req first through create_request()2 In process_header() in > state == H 2.1 call 'create_request' again with state set > to R, so main request gets created 2.2 call > 'ngx_http_upstream_send_request' manually to restart the > req-response cycle3 Because of 2.2 we get a process_header() > call in state R > Regards,Reetesh Obvious problem with the sequence: ngx_http_upstream_send_request() is static functions, and can't be called from a protocol module without modifications to ngx_http_upstream.c. I don't think that handshake can be implemented without modifications to upstream module. It's designed to implement request/response logic and doesn't assume any handshakes. -- Maxim Dounin http://nginx.org/en/donation.html From reeteshr at outlook.com Thu Dec 5 03:23:44 2013 From: reeteshr at outlook.com (Reetesh Ranjan) Date: Thu, 5 Dec 2013 08:53:44 +0530 Subject: How to implement handshake in an upstream module? In-Reply-To: <20131204223835.GZ93176@mdounin.ru> References: , <20131204223835.GZ93176@mdounin.ru> Message-ID: Hi, I meant making it non-static and using. This might be the modification we need to the upstream module, rather than introducing a 'handshake' handler formally. Was looking to see if there are any 'state machine' related catches in doing so. Thanks,Reetesh > Date: Thu, 5 Dec 2013 02:38:35 +0400 > From: mdounin at mdounin.ru > To: nginx-devel at nginx.org > Subject: Re: How to implement handshake in an upstream module? > > Hello! > > On Thu, Dec 05, 2013 at 02:39:28AM +0530, Reetesh Ranjan wrote: > > > Before I send my main request and process the response through > > create_request and process_header (and filter) callbacks, I need > > to have a short handshake with the upstream servers. It consists > > of a send() and a recv() from the upstream module. How to > > implement this? > > Would the following sequence work?States: H = do-handshake > > (initial state), R = do-request1 In state H, send handshake > > req first through create_request()2 In process_header() in > > state == H 2.1 call 'create_request' again with state set > > to R, so main request gets created 2.2 call > > 'ngx_http_upstream_send_request' manually to restart the > > req-response cycle3 Because of 2.2 we get a process_header() > > call in state R > > Regards,Reetesh > > Obvious problem with the sequence: > ngx_http_upstream_send_request() is static functions, and can't be > called from a protocol module without modifications to > ngx_http_upstream.c. > > I don't think that handshake can be implemented without > modifications to upstream module. It's designed to implement > request/response logic and doesn't assume any handshakes. > > -- > Maxim Dounin > http://nginx.org/en/donation.html > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From reeteshr at outlook.com Thu Dec 5 06:27:07 2013 From: reeteshr at outlook.com (Reetesh Ranjan) Date: Thu, 5 Dec 2013 11:57:07 +0530 Subject: How to implement handshake in an upstream module? In-Reply-To: References: , , <20131204223835.GZ93176@mdounin.ru>, Message-ID: Found a simple way. The sequence was: client server handshake ---> <--- handshake request ---> <--- response The handshake, the way it is in my specific case, is logically just an extension of the request and response streams. So just merged it into the same. client server handshake . request ---> <--- handshake . response So now things fit into the nginx state machine of upstream modules without any changes required. Regards,Reetesh From: reeteshr at outlook.com To: nginx-devel at nginx.org Subject: RE: How to implement handshake in an upstream module? Date: Thu, 5 Dec 2013 08:53:44 +0530 Hi, I meant making it non-static and using. This might be the modification we need to the upstream module, rather than introducing a 'handshake' handler formally. Was looking to see if there are any 'state machine' related catches in doing so. Thanks,Reetesh > Date: Thu, 5 Dec 2013 02:38:35 +0400 > From: mdounin at mdounin.ru > To: nginx-devel at nginx.org > Subject: Re: How to implement handshake in an upstream module? > > Hello! > > On Thu, Dec 05, 2013 at 02:39:28AM +0530, Reetesh Ranjan wrote: > > > Before I send my main request and process the response through > > create_request and process_header (and filter) callbacks, I need > > to have a short handshake with the upstream servers. It consists > > of a send() and a recv() from the upstream module. How to > > implement this? > > Would the following sequence work?States: H = do-handshake > > (initial state), R = do-request1 In state H, send handshake > > req first through create_request()2 In process_header() in > > state == H 2.1 call 'create_request' again with state set > > to R, so main request gets created 2.2 call > > 'ngx_http_upstream_send_request' manually to restart the > > req-response cycle3 Because of 2.2 we get a process_header() > > call in state R > > Regards,Reetesh > > Obvious problem with the sequence: > ngx_http_upstream_send_request() is static functions, and can't be > called from a protocol module without modifications to > ngx_http_upstream.c. > > I don't think that handshake can be implemented without > modifications to upstream module. It's designed to implement > request/response logic and doesn't assume any handshakes. > > -- > Maxim Dounin > http://nginx.org/en/donation.html > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel _______________________________________________ nginx-devel mailing list nginx-devel at nginx.org http://mailman.nginx.org/mailman/listinfo/nginx-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From reeteshr at outlook.com Thu Dec 5 10:40:55 2013 From: reeteshr at outlook.com (Reetesh Ranjan) Date: Thu, 5 Dec 2013 16:10:55 +0530 Subject: Sphinx2 Search Platform Upstream Module Pre-Release Message-ID: Hello! I have developed an upstream module for Sphinx2 search platform. It's available at: https://github.com/reeteshranjan/sphinx2-nginx-module It talks to Sphinx2 searchd component for performing search. It's in infant stage right now and more work is needed to get to production ready. I am working on it. In case any one is interested in working with me on this module, please let me know. If you are looking to play with it, it is built and installed like any other third party module using the source available at link provided above. Thanks,Reetesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From reeteshr at outlook.com Thu Dec 5 10:44:08 2013 From: reeteshr at outlook.com (Reetesh Ranjan) Date: Thu, 5 Dec 2013 16:14:08 +0530 Subject: Help on designing using multiple location/upstream modules In-Reply-To: References: , , , Message-ID: Hi! > Date: Sat, 30 Nov 2013 14:37:04 -0800 > Subject: Re: Help on designing using multiple location/upstream modules > From: agentzh at gmail.com > To: nginx-devel at nginx.org > > Hello! > > On Sat, Nov 30, 2013 at 11:05 AM, Reetesh Ranjan wrote: > > Thanks for the help! Going by the documentation of the lua-nginx-module on > > its subrequest handling it looks really promising for my use case. > > Great :) > > > I am > > currently writing the Sphinx2 upstream module. > > BTW, you could also build a lua-resty-sphinx2 library atop ngx_lua's > nonblocking cosocket API instead of writing an upstream C module. You > can check out the lua-resty-mysql or lua-resty-redis libraries for > examples: > > https://github.com/agentzh/lua-resty-mysql > > https://github.com/agentzh/lua-resty-redis > > The ngx_lua cosocket API is more flexible and much easier to use :) > That way you also don't have to use nginx subrequests at all ;) I was already midway writing the upstream module, and completed it. It was a great exercise anyway for looking into guts of nginx upstream state machine. :) I would look at the lua ones next. I am sure the learning obtained here and the eventual approach I had to take would be useful to build lua-resty-sphinx2. Looking forward to be part of the great openresty bundle! :) > > Would get back with questions > > in case I have any on using the lua-nginx-module for achieving what I need > > to do. > > > > You're recommended to post ngx_lua related questions to the > openresty-en mailing list: > > https://groups.google.com/group/openresty-en > > That way we can see your mails sooner rather later :) > > Best regards, > -agentzh > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From agentzh at gmail.com Thu Dec 5 21:23:55 2013 From: agentzh at gmail.com (Yichun Zhang (agentzh)) Date: Thu, 5 Dec 2013 13:23:55 -0800 Subject: How to implement handshake in an upstream module? In-Reply-To: References: <20131204223835.GZ93176@mdounin.ru> Message-ID: Hello! On Wed, Dec 4, 2013 at 10:27 PM, Reetesh Ranjan wrote: > The handshake, the way it is in my specific case, is logically just an > extension of the request and response streams. So just merged it into the > same. > > client server > handshake . request ---> > <--- handshake . response > > So now things fit into the nginx state machine of upstream modules without > any changes required. You're essentially doing pipelined requests here and you will run into the following limitation in the nginx core: http://mailman.nginx.org/pipermail/nginx-devel/2012-March/002040.html I ran into this when testing my ngx_redis2 module's pipelined request feature and my patch in that thread will help you :) Also, you may also want to deal with keepalive connections in your module in the future and you will want to save the handshake for reused connections (from the connection pool). Regards, -agentzh From reeteshr at outlook.com Fri Dec 6 06:26:41 2013 From: reeteshr at outlook.com (Reetesh Ranjan) Date: Fri, 6 Dec 2013 11:56:41 +0530 Subject: How to implement handshake in an upstream module? In-Reply-To: References: , <20131204223835.GZ93176@mdounin.ru>, , , Message-ID: Hi! > Date: Thu, 5 Dec 2013 13:23:55 -0800 > Subject: Re: How to implement handshake in an upstream module? > From: agentzh at gmail.com > To: nginx-devel at nginx.org > > You're essentially doing pipelined requests here and you will run into > the following limitation in the nginx core: > > http://mailman.nginx.org/pipermail/nginx-devel/2012-March/002040.html > > I ran into this when testing my ngx_redis2 module's pipelined request > feature and my patch in that thread will help you :) Thanks for the info. :) > Also, you may also want to deal with keepalive connections in your > module in the future and you will want to save the handshake for > reused connections (from the connection pool). Yeah, this is on the table for obvious performance reasons. I saw how you have put that upfront for redis, and that is really good. :) > > Regards, > -agentzh > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From ru at nginx.com Fri Dec 6 09:18:54 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Fri, 06 Dec 2013 09:18:54 +0000 Subject: [nginx] Configure: enable -Wconditional-uninitialized with clang. Message-ID: details: http://hg.nginx.org/nginx/rev/fedf777c6b24 branches: changeset: 5459:fedf777c6b24 user: Ruslan Ermilov date: Fri Dec 06 13:16:56 2013 +0400 description: Configure: enable -Wconditional-uninitialized with clang. diffstat: auto/cc/clang | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (11 lines): diff -r 4a71ef1aa36d -r fedf777c6b24 auto/cc/clang --- a/auto/cc/clang Wed Dec 04 23:13:13 2013 +0400 +++ b/auto/cc/clang Fri Dec 06 13:16:56 2013 +0400 @@ -82,6 +82,7 @@ fi # warnings CFLAGS="$CFLAGS $NGX_CLANG_OPT -Wall -Wextra -Wpointer-arith" +CFLAGS="$CFLAGS -Wconditional-uninitialized" #CFLAGS="$CFLAGS -Wmissing-prototypes" # we have a lot of unused function arguments From mdounin at mdounin.ru Fri Dec 6 16:00:10 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 06 Dec 2013 16:00:10 +0000 Subject: [nginx] Contrib: add vim scripts to contrib/ directory. Message-ID: details: http://hg.nginx.org/nginx/rev/f38043bd15f5 branches: changeset: 5460:f38043bd15f5 user: Evan Miller date: Wed Dec 04 14:45:44 2013 -0800 description: Contrib: add vim scripts to contrib/ directory. I've been maintaining these scripts independently for a while now, even though I'm no longer active in the community. Seems to me that contrib/ is a good long-term home for the scripts. diffstat: contrib/vim/ftdetect/nginx.vim | 4 + contrib/vim/indent/nginx.vim | 11 + contrib/vim/syntax/nginx.vim | 703 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 718 insertions(+), 0 deletions(-) diffs (truncated from 733 to 300 lines): diff --git a/contrib/vim/ftdetect/nginx.vim b/contrib/vim/ftdetect/nginx.vim new file mode 100644 --- /dev/null +++ b/contrib/vim/ftdetect/nginx.vim @@ -0,0 +1,4 @@ +au BufRead,BufNewFile *.nginx set ft=nginx +au BufRead,BufNewFile */etc/nginx/* set ft=nginx +au BufRead,BufNewFile */usr/local/nginx/conf/* set ft=nginx +au BufRead,BufNewFile nginx.conf set ft=nginx diff --git a/contrib/vim/indent/nginx.vim b/contrib/vim/indent/nginx.vim new file mode 100644 --- /dev/null +++ b/contrib/vim/indent/nginx.vim @@ -0,0 +1,11 @@ +if exists("b:did_indent") + finish +endif +let b:did_indent = 1 + +setlocal indentexpr= + +" cindent actually works for nginx' simple file structure +setlocal cindent +" Just make sure that the comments are not reset as defs would be. +setlocal cinkeys-=0# diff --git a/contrib/vim/syntax/nginx.vim b/contrib/vim/syntax/nginx.vim new file mode 100644 --- /dev/null +++ b/contrib/vim/syntax/nginx.vim @@ -0,0 +1,703 @@ +" Vim syntax file +" Language: nginx.conf + +if exists("b:current_syntax") + finish +end + +setlocal iskeyword+=. +setlocal iskeyword+=/ +setlocal iskeyword+=: + +syn match ngxVariable '\$\(\w\+\|{\w\+}\)' +syn match ngxVariableBlock '\$\(\w\+\|{\w\+}\)' contained +syn match ngxVariableString '\$\(\w\+\|{\w\+}\)' contained +syn region ngxBlock start=+^+ end=+{+ skip=+\${+ contains=ngxComment,ngxDirectiveBlock,ngxVariableBlock,ngxString oneline +syn region ngxString start=+\z(["']\)+ end=+\z1+ skip=+\\\\\|\\\z1+ contains=ngxVariableString +syn match ngxComment ' *#.*$' + +syn keyword ngxBoolean on +syn keyword ngxBoolean off + +syn keyword ngxDirectiveBlock http contained +syn keyword ngxDirectiveBlock mail contained +syn keyword ngxDirectiveBlock events contained +syn keyword ngxDirectiveBlock server contained +syn keyword ngxDirectiveBlock types contained +syn keyword ngxDirectiveBlock location contained +syn keyword ngxDirectiveBlock upstream contained +syn keyword ngxDirectiveBlock charset_map contained +syn keyword ngxDirectiveBlock limit_except contained +syn keyword ngxDirectiveBlock if contained +syn keyword ngxDirectiveBlock geo contained +syn keyword ngxDirectiveBlock map contained + +syn keyword ngxDirectiveImportant include +syn keyword ngxDirectiveImportant root +syn keyword ngxDirectiveImportant server +syn keyword ngxDirectiveImportant server_name +syn keyword ngxDirectiveImportant listen +syn keyword ngxDirectiveImportant internal +syn keyword ngxDirectiveImportant proxy_pass +syn keyword ngxDirectiveImportant memcached_pass +syn keyword ngxDirectiveImportant fastcgi_pass +syn keyword ngxDirectiveImportant try_files + +syn keyword ngxDirectiveControl break +syn keyword ngxDirectiveControl return +syn keyword ngxDirectiveControl rewrite +syn keyword ngxDirectiveControl set + +syn keyword ngxDirectiveError error_page +syn keyword ngxDirectiveError post_action + +syn keyword ngxDirectiveDeprecated connections +syn keyword ngxDirectiveDeprecated imap +syn keyword ngxDirectiveDeprecated open_file_cache_retest +syn keyword ngxDirectiveDeprecated optimize_server_names +syn keyword ngxDirectiveDeprecated satisfy_any + +syn keyword ngxDirective accept_mutex +syn keyword ngxDirective accept_mutex_delay +syn keyword ngxDirective access_log +syn keyword ngxDirective add_after_body +syn keyword ngxDirective add_before_body +syn keyword ngxDirective add_header +syn keyword ngxDirective addition_types +syn keyword ngxDirective aio +syn keyword ngxDirective alias +syn keyword ngxDirective allow +syn keyword ngxDirective ancient_browser +syn keyword ngxDirective ancient_browser_value +syn keyword ngxDirective auth_basic +syn keyword ngxDirective auth_basic_user_file +syn keyword ngxDirective auth_http +syn keyword ngxDirective auth_http_header +syn keyword ngxDirective auth_http_timeout +syn keyword ngxDirective autoindex +syn keyword ngxDirective autoindex_exact_size +syn keyword ngxDirective autoindex_localtime +syn keyword ngxDirective charset +syn keyword ngxDirective charset_types +syn keyword ngxDirective client_body_buffer_size +syn keyword ngxDirective client_body_in_file_only +syn keyword ngxDirective client_body_in_single_buffer +syn keyword ngxDirective client_body_temp_path +syn keyword ngxDirective client_body_timeout +syn keyword ngxDirective client_header_buffer_size +syn keyword ngxDirective client_header_timeout +syn keyword ngxDirective client_max_body_size +syn keyword ngxDirective connection_pool_size +syn keyword ngxDirective create_full_put_path +syn keyword ngxDirective daemon +syn keyword ngxDirective dav_access +syn keyword ngxDirective dav_methods +syn keyword ngxDirective debug_connection +syn keyword ngxDirective debug_points +syn keyword ngxDirective default_type +syn keyword ngxDirective degradation +syn keyword ngxDirective degrade +syn keyword ngxDirective deny +syn keyword ngxDirective devpoll_changes +syn keyword ngxDirective devpoll_events +syn keyword ngxDirective directio +syn keyword ngxDirective directio_alignment +syn keyword ngxDirective empty_gif +syn keyword ngxDirective env +syn keyword ngxDirective epoll_events +syn keyword ngxDirective error_log +syn keyword ngxDirective eventport_events +syn keyword ngxDirective expires +syn keyword ngxDirective fastcgi_bind +syn keyword ngxDirective fastcgi_buffer_size +syn keyword ngxDirective fastcgi_buffers +syn keyword ngxDirective fastcgi_busy_buffers_size +syn keyword ngxDirective fastcgi_cache +syn keyword ngxDirective fastcgi_cache_key +syn keyword ngxDirective fastcgi_cache_methods +syn keyword ngxDirective fastcgi_cache_min_uses +syn keyword ngxDirective fastcgi_cache_path +syn keyword ngxDirective fastcgi_cache_use_stale +syn keyword ngxDirective fastcgi_cache_valid +syn keyword ngxDirective fastcgi_catch_stderr +syn keyword ngxDirective fastcgi_connect_timeout +syn keyword ngxDirective fastcgi_hide_header +syn keyword ngxDirective fastcgi_ignore_client_abort +syn keyword ngxDirective fastcgi_ignore_headers +syn keyword ngxDirective fastcgi_index +syn keyword ngxDirective fastcgi_intercept_errors +syn keyword ngxDirective fastcgi_max_temp_file_size +syn keyword ngxDirective fastcgi_next_upstream +syn keyword ngxDirective fastcgi_param +syn keyword ngxDirective fastcgi_pass_header +syn keyword ngxDirective fastcgi_pass_request_body +syn keyword ngxDirective fastcgi_pass_request_headers +syn keyword ngxDirective fastcgi_read_timeout +syn keyword ngxDirective fastcgi_send_lowat +syn keyword ngxDirective fastcgi_send_timeout +syn keyword ngxDirective fastcgi_split_path_info +syn keyword ngxDirective fastcgi_store +syn keyword ngxDirective fastcgi_store_access +syn keyword ngxDirective fastcgi_temp_file_write_size +syn keyword ngxDirective fastcgi_temp_path +syn keyword ngxDirective fastcgi_upstream_fail_timeout +syn keyword ngxDirective fastcgi_upstream_max_fails +syn keyword ngxDirective flv +syn keyword ngxDirective geoip_city +syn keyword ngxDirective geoip_country +syn keyword ngxDirective google_perftools_profiles +syn keyword ngxDirective gzip +syn keyword ngxDirective gzip_buffers +syn keyword ngxDirective gzip_comp_level +syn keyword ngxDirective gzip_disable +syn keyword ngxDirective gzip_hash +syn keyword ngxDirective gzip_http_version +syn keyword ngxDirective gzip_min_length +syn keyword ngxDirective gzip_no_buffer +syn keyword ngxDirective gzip_proxied +syn keyword ngxDirective gzip_static +syn keyword ngxDirective gzip_types +syn keyword ngxDirective gzip_vary +syn keyword ngxDirective gzip_window +syn keyword ngxDirective if_modified_since +syn keyword ngxDirective ignore_invalid_headers +syn keyword ngxDirective image_filter +syn keyword ngxDirective image_filter_buffer +syn keyword ngxDirective image_filter_jpeg_quality +syn keyword ngxDirective image_filter_transparency +syn keyword ngxDirective imap_auth +syn keyword ngxDirective imap_capabilities +syn keyword ngxDirective imap_client_buffer +syn keyword ngxDirective index +syn keyword ngxDirective ip_hash +syn keyword ngxDirective keepalive_requests +syn keyword ngxDirective keepalive_timeout +syn keyword ngxDirective kqueue_changes +syn keyword ngxDirective kqueue_events +syn keyword ngxDirective large_client_header_buffers +syn keyword ngxDirective limit_conn +syn keyword ngxDirective limit_conn_log_level +syn keyword ngxDirective limit_rate +syn keyword ngxDirective limit_rate_after +syn keyword ngxDirective limit_req +syn keyword ngxDirective limit_req_log_level +syn keyword ngxDirective limit_req_zone +syn keyword ngxDirective limit_zone +syn keyword ngxDirective lingering_time +syn keyword ngxDirective lingering_timeout +syn keyword ngxDirective lock_file +syn keyword ngxDirective log_format +syn keyword ngxDirective log_not_found +syn keyword ngxDirective log_subrequest +syn keyword ngxDirective map_hash_bucket_size +syn keyword ngxDirective map_hash_max_size +syn keyword ngxDirective master_process +syn keyword ngxDirective memcached_bind +syn keyword ngxDirective memcached_buffer_size +syn keyword ngxDirective memcached_connect_timeout +syn keyword ngxDirective memcached_next_upstream +syn keyword ngxDirective memcached_read_timeout +syn keyword ngxDirective memcached_send_timeout +syn keyword ngxDirective memcached_upstream_fail_timeout +syn keyword ngxDirective memcached_upstream_max_fails +syn keyword ngxDirective merge_slashes +syn keyword ngxDirective min_delete_depth +syn keyword ngxDirective modern_browser +syn keyword ngxDirective modern_browser_value +syn keyword ngxDirective msie_padding +syn keyword ngxDirective msie_refresh +syn keyword ngxDirective multi_accept +syn keyword ngxDirective open_file_cache +syn keyword ngxDirective open_file_cache_errors +syn keyword ngxDirective open_file_cache_events +syn keyword ngxDirective open_file_cache_min_uses +syn keyword ngxDirective open_file_cache_valid +syn keyword ngxDirective open_log_file_cache +syn keyword ngxDirective output_buffers +syn keyword ngxDirective override_charset +syn keyword ngxDirective perl +syn keyword ngxDirective perl_modules +syn keyword ngxDirective perl_require +syn keyword ngxDirective perl_set +syn keyword ngxDirective pid +syn keyword ngxDirective pop3_auth +syn keyword ngxDirective pop3_capabilities +syn keyword ngxDirective port_in_redirect +syn keyword ngxDirective postpone_gzipping +syn keyword ngxDirective postpone_output +syn keyword ngxDirective protocol +syn keyword ngxDirective proxy +syn keyword ngxDirective proxy_bind +syn keyword ngxDirective proxy_buffer +syn keyword ngxDirective proxy_buffer_size +syn keyword ngxDirective proxy_buffering +syn keyword ngxDirective proxy_buffers +syn keyword ngxDirective proxy_busy_buffers_size +syn keyword ngxDirective proxy_cache +syn keyword ngxDirective proxy_cache_key +syn keyword ngxDirective proxy_cache_methods +syn keyword ngxDirective proxy_cache_min_uses +syn keyword ngxDirective proxy_cache_path +syn keyword ngxDirective proxy_cache_use_stale +syn keyword ngxDirective proxy_cache_valid +syn keyword ngxDirective proxy_connect_timeout +syn keyword ngxDirective proxy_headers_hash_bucket_size +syn keyword ngxDirective proxy_headers_hash_max_size +syn keyword ngxDirective proxy_hide_header +syn keyword ngxDirective proxy_ignore_client_abort +syn keyword ngxDirective proxy_ignore_headers +syn keyword ngxDirective proxy_intercept_errors +syn keyword ngxDirective proxy_max_temp_file_size +syn keyword ngxDirective proxy_method +syn keyword ngxDirective proxy_next_upstream +syn keyword ngxDirective proxy_pass_error_message +syn keyword ngxDirective proxy_pass_header +syn keyword ngxDirective proxy_pass_request_body +syn keyword ngxDirective proxy_pass_request_headers +syn keyword ngxDirective proxy_read_timeout +syn keyword ngxDirective proxy_redirect +syn keyword ngxDirective proxy_send_lowat +syn keyword ngxDirective proxy_send_timeout +syn keyword ngxDirective proxy_set_body +syn keyword ngxDirective proxy_set_header +syn keyword ngxDirective proxy_ssl_session_reuse +syn keyword ngxDirective proxy_store +syn keyword ngxDirective proxy_store_access +syn keyword ngxDirective proxy_temp_file_write_size +syn keyword ngxDirective proxy_temp_path +syn keyword ngxDirective proxy_timeout +syn keyword ngxDirective proxy_upstream_fail_timeout +syn keyword ngxDirective proxy_upstream_max_fails From mdounin at mdounin.ru Fri Dec 6 16:00:12 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 06 Dec 2013 16:00:12 +0000 Subject: [nginx] Contrib: added vim to README. Message-ID: details: http://hg.nginx.org/nginx/rev/1bdf906bba99 branches: changeset: 5461:1bdf906bba99 user: Maxim Dounin date: Fri Dec 06 19:58:27 2013 +0400 description: Contrib: added vim to README. diffstat: contrib/README | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diffs (13 lines): diff --git a/contrib/README b/contrib/README --- a/contrib/README +++ b/contrib/README @@ -13,3 +13,9 @@ unicode2nginx by Maxim Dounin configuration file format. Two generated full maps for windows-1251 and koi8-r. + +vim by Evan Miller + + Syntax highlighting of nginx configuration for vim, to be + placed into ~/.vim/. + From sarah.novotny at gmail.com Fri Dec 6 16:19:44 2013 From: sarah.novotny at gmail.com (sarah novotny) Date: Fri, 6 Dec 2013 17:19:44 +0100 Subject: [nginx] Contrib: added vim to README. In-Reply-To: References: Message-ID: <895403B6-3B58-4839-98C0-53CF87FF7317@gmail.com> fun! sarah > On Dec 6, 2013, at 5:00 PM, Maxim Dounin wrote: > > details: http://hg.nginx.org/nginx/rev/1bdf906bba99 > branches: > changeset: 5461:1bdf906bba99 > user: Maxim Dounin > date: Fri Dec 06 19:58:27 2013 +0400 > description: > Contrib: added vim to README. > > diffstat: > > contrib/README | 6 ++++++ > 1 files changed, 6 insertions(+), 0 deletions(-) > > diffs (13 lines): > > diff --git a/contrib/README b/contrib/README > --- a/contrib/README > +++ b/contrib/README > @@ -13,3 +13,9 @@ unicode2nginx by Maxim Dounin > configuration file format. > Two generated full maps for windows-1251 and koi8-r. > > + > +vim by Evan Miller > + > + Syntax highlighting of nginx configuration for vim, to be > + placed into ~/.vim/. > + > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel From piotr at cloudflare.com Fri Dec 6 23:45:31 2013 From: piotr at cloudflare.com (Piotr Sikora) Date: Fri, 6 Dec 2013 15:45:31 -0800 Subject: [PATCH] Fixed incorrect ngx_str_set() usage, broken in c82b2e020b9f. Message-ID: # HG changeset patch # User Piotr Sikora # Date 1386373432 28800 # Fri Dec 06 15:43:52 2013 -0800 # Node ID 2f544de052c59ea14094473a7089e27d661b6bc8 # Parent 1bdf906bba99b4c03d306dd8f2756d4d8ffa4a13 Fixed incorrect ngx_str_set() usage, broken in c82b2e020b9f. Found by Coverity Scan CID 113525. Signed-off-by: Piotr Sikora diff -r 1bdf906bba99 -r 2f544de052c5 src/http/modules/ngx_http_uwsgi_module.c --- a/src/http/modules/ngx_http_uwsgi_module.c Fri Dec 06 19:58:27 2013 +0400 +++ b/src/http/modules/ngx_http_uwsgi_module.c Fri Dec 06 15:43:52 2013 -0800 @@ -603,7 +603,7 @@ ngx_http_uwsgi_eval(ngx_http_request_t * url.url.len -= add; } else { - ngx_str_set(&u->schema, (u_char *) "uwsgi://"); + ngx_str_set(&u->schema, "uwsgi://"); } url.no_resolve = 1; From piotr at cloudflare.com Sat Dec 7 00:01:52 2013 From: piotr at cloudflare.com (Piotr Sikora) Date: Fri, 6 Dec 2013 16:01:52 -0800 Subject: [PATCH] Fixed incorrect ngx_str_set() usage, broken in c82b2e020b9f. In-Reply-To: References: Message-ID: Damn it, I've lost one digit in Scan CID. # HG changeset patch # User Piotr Sikora # Date 1386374449 28800 # Fri Dec 06 16:00:49 2013 -0800 # Node ID 58716fd3bd2d63c93b0c04fa121232b7126e724b # Parent 1bdf906bba99b4c03d306dd8f2756d4d8ffa4a13 Fixed incorrect ngx_str_set() usage, broken in c82b2e020b9f. Found by Coverity Scan CID 1135525. Signed-off-by: Piotr Sikora diff -r 1bdf906bba99 -r 58716fd3bd2d src/http/modules/ngx_http_uwsgi_module.c --- a/src/http/modules/ngx_http_uwsgi_module.c Fri Dec 06 19:58:27 2013 +0400 +++ b/src/http/modules/ngx_http_uwsgi_module.c Fri Dec 06 16:00:49 2013 -0800 @@ -603,7 +603,7 @@ ngx_http_uwsgi_eval(ngx_http_request_t * url.url.len -= add; } else { - ngx_str_set(&u->schema, (u_char *) "uwsgi://"); + ngx_str_set(&u->schema, "uwsgi://"); } url.no_resolve = 1; From mdounin at mdounin.ru Sat Dec 7 13:29:08 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Sat, 07 Dec 2013 13:29:08 +0000 Subject: [nginx] Fixed incorrect ngx_str_set() usage, broken in c82b2e020... Message-ID: details: http://hg.nginx.org/nginx/rev/58716fd3bd2d branches: changeset: 5462:58716fd3bd2d user: Piotr Sikora date: Fri Dec 06 16:00:49 2013 -0800 description: Fixed incorrect ngx_str_set() usage, broken in c82b2e020b9f. Found by Coverity Scan CID 1135525. Signed-off-by: Piotr Sikora diffstat: src/http/modules/ngx_http_uwsgi_module.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff --git a/src/http/modules/ngx_http_uwsgi_module.c b/src/http/modules/ngx_http_uwsgi_module.c --- a/src/http/modules/ngx_http_uwsgi_module.c +++ b/src/http/modules/ngx_http_uwsgi_module.c @@ -603,7 +603,7 @@ ngx_http_uwsgi_eval(ngx_http_request_t * url.url.len -= add; } else { - ngx_str_set(&u->schema, (u_char *) "uwsgi://"); + ngx_str_set(&u->schema, "uwsgi://"); } url.no_resolve = 1; From mdounin at mdounin.ru Sat Dec 7 13:29:37 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Sat, 7 Dec 2013 17:29:37 +0400 Subject: [PATCH] Fixed incorrect ngx_str_set() usage, broken in c82b2e020b9f. In-Reply-To: References: Message-ID: <20131207132937.GT95113@mdounin.ru> Hello! On Fri, Dec 06, 2013 at 04:01:52PM -0800, Piotr Sikora wrote: > Damn it, I've lost one digit in Scan CID. > > # HG changeset patch > # User Piotr Sikora > # Date 1386374449 28800 > # Fri Dec 06 16:00:49 2013 -0800 > # Node ID 58716fd3bd2d63c93b0c04fa121232b7126e724b > # Parent 1bdf906bba99b4c03d306dd8f2756d4d8ffa4a13 > Fixed incorrect ngx_str_set() usage, broken in c82b2e020b9f. > > Found by Coverity Scan CID 1135525. > > Signed-off-by: Piotr Sikora > > diff -r 1bdf906bba99 -r 58716fd3bd2d src/http/modules/ngx_http_uwsgi_module.c > --- a/src/http/modules/ngx_http_uwsgi_module.c Fri Dec 06 19:58:27 2013 +0400 > +++ b/src/http/modules/ngx_http_uwsgi_module.c Fri Dec 06 16:00:49 2013 -0800 > @@ -603,7 +603,7 @@ ngx_http_uwsgi_eval(ngx_http_request_t * > url.url.len -= add; > > } else { > - ngx_str_set(&u->schema, (u_char *) "uwsgi://"); > + ngx_str_set(&u->schema, "uwsgi://"); > } > > url.no_resolve = 1; Committed, thanks. -- Maxim Dounin http://nginx.org/en/donation.html From rommer at activecloud.com Sat Dec 7 23:40:26 2013 From: rommer at activecloud.com (Raman Shishniou) Date: Sun, 8 Dec 2013 02:40:26 +0300 Subject: [PATCH] Make ngx_http_parse_unsafe_uri() to be able to unescape uri Message-ID: <52A3B1EA.3060203@activecloud.com> # HG changeset patch # User Raman Shishniou # Date 1386459301 -10800 # Node ID 54f3670e04e82e00aa424d9773868749026bc693 # Parent 58716fd3bd2d63c93b0c04fa121232b7126e724b Make ngx_http_parse_unsafe_uri() to be able to unescape uri It makes possible to use an escaped uri with "X-Accel-Redirect" header in http_upstream module and "Destination" header http_dav module. No need to unescape uri in http_ssi_filter module any more. diff --git a/src/http/modules/ngx_http_dav_module.c b/src/http/modules/ngx_http_dav_module.c --- a/src/http/modules/ngx_http_dav_module.c +++ b/src/http/modules/ngx_http_dav_module.c @@ -517,7 +517,7 @@ size_t len, root; ngx_err_t err; ngx_int_t rc, depth; - ngx_uint_t overwrite, slash, dir, flags; + ngx_uint_t overwrite, slash, dir; ngx_str_t path, uri, duri, args; ngx_tree_ctx_t tree; ngx_copy_file_t cf; @@ -604,9 +604,8 @@ duri.len = last - p; duri.data = p; - flags = 0; - if (ngx_http_parse_unsafe_uri(r, &duri, &args, &flags) != NGX_OK) { + if (ngx_http_parse_unsafe_uri(r, &duri, &args, 0) != NGX_OK) { goto invalid_destination; } diff --git a/src/http/modules/ngx_http_ssi_filter_module.c b/src/http/modules/ngx_http_ssi_filter_module.c --- a/src/http/modules/ngx_http_ssi_filter_module.c +++ b/src/http/modules/ngx_http_ssi_filter_module.c @@ -1982,8 +1982,6 @@ ngx_http_ssi_include(ngx_http_request_t *r, ngx_http_ssi_ctx_t *ctx, ngx_str_t **params) { - u_char *dst, *src; - size_t len; ngx_int_t rc, key; ngx_str_t *uri, *file, *wait, *set, *stub, args; ngx_buf_t *b; @@ -2054,25 +2052,13 @@ return rc; } - dst = uri->data; - src = uri->data; - - ngx_unescape_uri(&dst, &src, uri->len, NGX_UNESCAPE_URI); - - len = (uri->data + uri->len) - src; - if (len) { - dst = ngx_movemem(dst, src, len); - } - - uri->len = dst - uri->data; - ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "ssi include: \"%V\"", uri); ngx_str_null(&args); - flags = NGX_HTTP_LOG_UNSAFE; - - if (ngx_http_parse_unsafe_uri(r, uri, &args, &flags) != NGX_OK) { + + if (ngx_http_parse_unsafe_uri(r, uri, &args, + NGX_HTTP_LOG_UNSAFE) != NGX_OK) { return NGX_HTTP_SSI_ERROR; } @@ -2147,6 +2133,8 @@ } } + flags = 0; + if (wait) { flags |= NGX_HTTP_SUBREQUEST_WAITED; } diff --git a/src/http/ngx_http.h b/src/http/ngx_http.h --- a/src/http/ngx_http.h +++ b/src/http/ngx_http.h @@ -100,7 +100,7 @@ ngx_int_t ngx_http_parse_status_line(ngx_http_request_t *r, ngx_buf_t *b, ngx_http_status_t *status); ngx_int_t ngx_http_parse_unsafe_uri(ngx_http_request_t *r, ngx_str_t *uri, - ngx_str_t *args, ngx_uint_t *flags); + ngx_str_t *args, ngx_uint_t flags); ngx_int_t ngx_http_parse_header_line(ngx_http_request_t *r, ngx_buf_t *b, ngx_uint_t allow_underscores); ngx_int_t ngx_http_parse_multi_header_lines(ngx_array_t *headers, diff --git a/src/http/ngx_http_parse.c b/src/http/ngx_http_parse.c --- a/src/http/ngx_http_parse.c +++ b/src/http/ngx_http_parse.c @@ -1778,57 +1778,233 @@ ngx_int_t ngx_http_parse_unsafe_uri(ngx_http_request_t *r, ngx_str_t *uri, - ngx_str_t *args, ngx_uint_t *flags) + ngx_str_t *args, ngx_uint_t flags) { - u_char ch, *p; - size_t len; - + u_char *src, *dst, *newuri, ch, c, decoded; + size_t len; + enum { + sw_usual = 0, + sw_quoted, + sw_quoted_second + } state; + + dst = NULL; + newuri = NULL; + src = uri->data; len = uri->len; - p = uri->data; - - if (len == 0 || p[0] == '?') { + + state = 0; + decoded = 0; + + if (len == 0 || src[0] == '?') { goto unsafe; } - if (p[0] == '.' && len == 3 && p[1] == '.' && (ngx_path_separator(p[2]))) { - goto unsafe; - } - - for ( /* void */ ; len; len--) { - - ch = *p++; - - if (usual[ch >> 5] & (1 << (ch & 0x1f))) { - continue; - } - - if (ch == '?') { - args->len = len - 1; - args->data = p; - uri->len -= len; - - return NGX_OK; - } - - if (ch == '\0') { - goto unsafe; - } - - if (ngx_path_separator(ch) && len > 2) { - - /* detect "/../" */ - - if (p[0] == '.' && p[1] == '.' && ngx_path_separator(p[2])) { + while (len--) { + + ch = *src++; + + switch (state) { + case sw_usual: + if (usual[ch >> 5] & (1 << (ch & 0x1f))) { + if (dst != NULL) { + *dst++ = ch; + } + break; + } + + if (ch == '\0') { goto unsafe; } + + if (ch == '?') { + goto args; + } + + if (ch == '%') { + if (dst == NULL) { + newuri = ngx_pnalloc(r->pool, uri->len); + if (newuri == NULL) { + return NGX_ERROR; + } + if (src - 1 > uri->data) { + ngx_memcpy(newuri, uri->data, src - uri->data - 1); + dst = newuri + (src - uri->data - 1); + } else { + dst = newuri; + } + } + state = sw_quoted; + break; + } + + /* detect "../" from begin or "/../" in the middle */ + if (ngx_path_separator(ch)) { + if (dst == NULL) { + + if (src - 2 == uri->data && + *(src - 2) == '.' && *(src - 1) == '.') { + goto unsafe; + } + + if (src - 2 > uri->data && + ngx_path_separator(*(dst - 3)) && + *(src - 2) == '.' && *(src - 1) == '.') { + goto unsafe; + } + + } else { + + if (dst - 2 == newuri && + newuri[0] == '.' && newuri[1] == '.') { + goto unsafe; + } + + if (dst - 2 > newuri && + ngx_path_separator(*(dst - 3)) && + *(dst - 2) == '.' && *(dst - 1) == '.') { + goto unsafe; + } + + } + } + + if (dst != NULL) { + *dst++ = ch; + } + break; + + case sw_quoted: + if (ch >= '0' && ch <= '9') { + decoded = (u_char) (ch - '0'); + state = sw_quoted_second; + break; + } + + c = (u_char) (ch | 0x20); + if (c >= 'a' && c <= 'f') { + decoded = (u_char) (c - 'a' + 10); + state = sw_quoted_second; + break; + } + + if (ch == '\0') { + goto unsafe; + } + + if (ch == '%') { + *dst++ = '%'; + break; + } + + if (ch == '?') { + *dst++ = '%'; + goto args; + } + + *dst++ = '%'; + *dst++ = ch; + state = sw_usual; + break; + + case sw_quoted_second: + + state = sw_usual; + + if (ch >= '0' && ch <= '9') { + ch = (u_char) ((decoded << 4) + ch - '0'); + + if (ch == '\0') { + goto unsafe; + } + + *dst++ = ch; + break; + } + + c = (u_char) (ch | 0x20); + if (c >= 'a' && c <= 'f') { + ch = (u_char) ((decoded << 4) + c - 'a' + 10); + + /* detect "../" from begin or "/../" in the middle */ + if (ngx_path_separator(ch)) { + + if (dst - 2 == newuri && + newuri[0] == '.' && newuri[1] == '.') { + goto unsafe; + } + + if (dst - 2 > newuri && + ngx_path_separator(*(dst - 3)) && + *(dst - 2) == '.' && *(dst - 1) == '.') { + goto unsafe; + } + } + + *dst++ = ch; + break; + } + + if (ch == '\0') { + goto unsafe; + } + + if (ch == '%') { + *dst++ = '%'; + *dst++ = *(src - 2); + state = sw_quoted; + break; + } + + if (ch == '?') { + *dst++ = '%'; + *dst++ = *(src - 2); + goto args; + } + + *dst++ = '%'; + *dst++ = *(src - 2); + *dst++ = ch; + break; } } + switch (state) { + case sw_usual: + break; + case sw_quoted: + *dst++ = '%'; + break; + case sw_quoted_second: + *dst++ = '%'; + *dst++ = *(src - 1); + break; + } + + if (dst != NULL) { + uri->len = dst - newuri; + uri->data = newuri; + } + return NGX_OK; +args: + + args->len = len; + args->data = src; + + if (dst != NULL) { + uri->len = dst - newuri; + uri->data = newuri; + } else { + uri->len = src - uri->data; + } + + return NGX_OK; + unsafe: - if (*flags & NGX_HTTP_LOG_UNSAFE) { + if (flags & NGX_HTTP_LOG_UNSAFE) { ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, "unsafe URI \"%V\" was detected", uri); } diff --git a/src/http/ngx_http_upstream.c b/src/http/ngx_http_upstream.c --- a/src/http/ngx_http_upstream.c +++ b/src/http/ngx_http_upstream.c @@ -1994,7 +1994,7 @@ ngx_http_upstream_process_headers(ngx_http_request_t *r, ngx_http_upstream_t *u) { ngx_str_t *uri, args; - ngx_uint_t i, flags; + ngx_uint_t i; ngx_list_part_t *part; ngx_table_elt_t *h; ngx_http_upstream_header_t *hh; @@ -2036,9 +2036,9 @@ uri = &u->headers_in.x_accel_redirect->value; ngx_str_null(&args); - flags = NGX_HTTP_LOG_UNSAFE; - - if (ngx_http_parse_unsafe_uri(r, uri, &args, &flags) != NGX_OK) { + + if (ngx_http_parse_unsafe_uri(r, uri, &args, + NGX_HTTP_LOG_UNSAFE) != NGX_OK) { ngx_http_finalize_request(r, NGX_HTTP_NOT_FOUND); return NGX_DONE; } From ru at nginx.com Mon Dec 9 06:17:55 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Mon, 09 Dec 2013 06:17:55 +0000 Subject: [nginx] Core: keep the length of the local sockaddr. Message-ID: details: http://hg.nginx.org/nginx/rev/1ab1cf63f885 branches: changeset: 5463:1ab1cf63f885 user: Ruslan Ermilov date: Mon Dec 09 10:14:51 2013 +0400 description: Core: keep the length of the local sockaddr. diffstat: src/core/ngx_connection.c | 11 ++++++++--- src/core/ngx_connection.h | 1 + src/event/ngx_event_accept.c | 1 + src/event/ngx_event_acceptex.c | 3 +-- 4 files changed, 11 insertions(+), 5 deletions(-) diffs (88 lines): diff -r 58716fd3bd2d -r 1ab1cf63f885 src/core/ngx_connection.c --- a/src/core/ngx_connection.c Fri Dec 06 16:00:49 2013 -0800 +++ b/src/core/ngx_connection.c Mon Dec 09 10:14:51 2013 +0400 @@ -1078,12 +1078,15 @@ ngx_connection_local_sockaddr(ngx_connec struct sockaddr_in6 *sin6; #endif + if (c->local_socklen == 0) { + return NGX_ERROR; + } + switch (c->local_sockaddr->sa_family) { #if (NGX_HAVE_INET6) case AF_INET6: sin6 = (struct sockaddr_in6 *) c->local_sockaddr; - len = sizeof(struct sockaddr_in6); for (addr = 0, i = 0; addr == 0 && i < 16; i++) { addr |= sin6->sin6_addr.s6_addr[i]; @@ -1094,7 +1097,6 @@ ngx_connection_local_sockaddr(ngx_connec default: /* AF_INET */ sin = (struct sockaddr_in *) c->local_sockaddr; - len = sizeof(struct sockaddr_in); addr = sin->sin_addr.s_addr; break; } @@ -1114,13 +1116,16 @@ ngx_connection_local_sockaddr(ngx_connec } ngx_memcpy(c->local_sockaddr, &sa, len); + + c->local_socklen = len; } if (s == NULL) { return NGX_OK; } - s->len = ngx_sock_ntop(c->local_sockaddr, len, s->data, s->len, port); + s->len = ngx_sock_ntop(c->local_sockaddr, c->local_socklen, + s->data, s->len, port); return NGX_OK; } diff -r 58716fd3bd2d -r 1ab1cf63f885 src/core/ngx_connection.h --- a/src/core/ngx_connection.h Fri Dec 06 16:00:49 2013 -0800 +++ b/src/core/ngx_connection.h Mon Dec 09 10:14:51 2013 +0400 @@ -143,6 +143,7 @@ struct ngx_connection_s { #endif struct sockaddr *local_sockaddr; + socklen_t local_socklen; ngx_buf_t *buffer; diff -r 58716fd3bd2d -r 1ab1cf63f885 src/event/ngx_event_accept.c --- a/src/event/ngx_event_accept.c Fri Dec 06 16:00:49 2013 -0800 +++ b/src/event/ngx_event_accept.c Mon Dec 09 10:14:51 2013 +0400 @@ -212,6 +212,7 @@ ngx_event_accept(ngx_event_t *ev) c->socklen = socklen; c->listening = ls; c->local_sockaddr = ls->sockaddr; + c->local_socklen = ls->socklen; c->unexpected_eof = 1; diff -r 58716fd3bd2d -r 1ab1cf63f885 src/event/ngx_event_acceptex.c --- a/src/event/ngx_event_acceptex.c Fri Dec 06 16:00:49 2013 -0800 +++ b/src/event/ngx_event_acceptex.c Mon Dec 09 10:14:51 2013 +0400 @@ -18,7 +18,6 @@ ngx_event_acceptex(ngx_event_t *rev) { ngx_listening_t *ls; ngx_connection_t *c; - socklen_t len; c = rev->data; ls = c->listening; @@ -50,7 +49,7 @@ ngx_event_acceptex(ngx_event_t *rev) ls->post_accept_buffer_size, ls->socklen + 16, ls->socklen + 16, - &c->local_sockaddr, &len, + &c->local_sockaddr, &c->local_socklen, &c->sockaddr, &c->socklen); if (ls->post_accept_buffer_size) { From ru at nginx.com Mon Dec 9 06:17:57 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Mon, 09 Dec 2013 06:17:57 +0000 Subject: [nginx] Fixed handling of UNIX-domain sockets. Message-ID: details: http://hg.nginx.org/nginx/rev/30e806b8636a branches: changeset: 5464:30e806b8636a user: Ruslan Ermilov date: Mon Dec 09 10:16:44 2013 +0400 description: Fixed handling of UNIX-domain sockets. When evaluating $local_port, $server_port, and $server_addr, UNIX-domain sockets were mistakenly interpreted as IPv4 sockets. diffstat: src/core/ngx_connection.c | 6 ++++++ src/http/ngx_http_variables.c | 12 ++++++++++++ 2 files changed, 18 insertions(+), 0 deletions(-) diffs (45 lines): diff -r 1ab1cf63f885 -r 30e806b8636a src/core/ngx_connection.c --- a/src/core/ngx_connection.c Mon Dec 09 10:14:51 2013 +0400 +++ b/src/core/ngx_connection.c Mon Dec 09 10:16:44 2013 +0400 @@ -1095,6 +1095,12 @@ ngx_connection_local_sockaddr(ngx_connec break; #endif +#if (NGX_HAVE_UNIX_DOMAIN) + case AF_UNIX: + addr = 1; + break; +#endif + default: /* AF_INET */ sin = (struct sockaddr_in *) c->local_sockaddr; addr = sin->sin_addr.s_addr; diff -r 1ab1cf63f885 -r 30e806b8636a src/http/ngx_http_variables.c --- a/src/http/ngx_http_variables.c Mon Dec 09 10:14:51 2013 +0400 +++ b/src/http/ngx_http_variables.c Mon Dec 09 10:16:44 2013 +0400 @@ -1185,6 +1185,12 @@ ngx_http_variable_remote_port(ngx_http_r break; #endif +#if (NGX_HAVE_UNIX_DOMAIN) + case AF_UNIX: + port = 0; + break; +#endif + default: /* AF_INET */ sin = (struct sockaddr_in *) r->connection->sockaddr; port = ntohs(sin->sin_port); @@ -1263,6 +1269,12 @@ ngx_http_variable_server_port(ngx_http_r break; #endif +#if (NGX_HAVE_UNIX_DOMAIN) + case AF_UNIX: + port = 0; + break; +#endif + default: /* AF_INET */ sin = (struct sockaddr_in *) r->connection->local_sockaddr; port = ntohs(sin->sin_port); From ru at nginx.com Mon Dec 9 10:29:29 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Mon, 9 Dec 2013 14:29:29 +0400 Subject: IPv6 & IPv4 backend with proxy_bind In-Reply-To: References: <20131118114554.GB41579@mdounin.ru> <20131118132740.GH41579@mdounin.ru> <20131122132642.GO56821@lo0.su> Message-ID: <20131209102929.GC59059@lo0.su> On Sat, Nov 23, 2013 at 12:15:28PM +1030, SplitIce wrote: > Attached is the patch, > > This is the first time I have created a variable or really done anything > inside the http request processing flow so feel free to let me know if > there is a better way to do something or if I have any edge cases. > > This patch provides a $upstream_connecting variable which contains the IP > address and port of the upstream being connected. If there is no upstream, > it will return "-" my understanding is this may happen if the upstream is > DNS resolved (untested). There may be a better way of doing this? > > This should be used in a config like the following - > map $upstream_connecting $test { > ~^93\.184\.216\.119\: 192.168.2.40; > ~^192\.168\.2\.([0-9]+)\: 192.168.2.40; > } > > proxy_bind $test; I took a different approach. I've made "local" a peer's method, so the computation of the local address is delayed until it's actually needed. By that time, the peer address is already known. I've also patched "map" so it creates non-cacheable variables. This way, $upstream_peer_addr can be mapped into the local address: map $upstream_peer_addr $bind_addr { 127.0.0.1:8001 127.0.0.1; 127.0.0.1:8002 127.0.0.1; [::1]:8003 ::1; } server { ... location / { proxy_pass ...; proxy_bind $bind_addr; ... } } diff --git a/src/event/ngx_event_connect.c b/src/event/ngx_event_connect.c --- a/src/event/ngx_event_connect.c +++ b/src/event/ngx_event_connect.c @@ -17,6 +17,7 @@ ngx_event_connect_peer(ngx_peer_connecti int rc; ngx_int_t event; ngx_err_t err; + ngx_addr_t *local; ngx_uint_t level; ngx_socket_t s; ngx_event_t *rev, *wev; @@ -67,9 +68,13 @@ ngx_event_connect_peer(ngx_peer_connecti } if (pc->local) { - if (bind(s, pc->local->sockaddr, pc->local->socklen) == -1) { + local = pc->local(pc, pc->data); + + if (local != NULL + && bind(s, local->sockaddr, local->socklen) == -1) + { ngx_log_error(NGX_LOG_CRIT, pc->log, ngx_socket_errno, - "bind(%V) failed", &pc->local->name); + "bind(%V) failed", &local->name); goto failed; } diff --git a/src/event/ngx_event_connect.h b/src/event/ngx_event_connect.h --- a/src/event/ngx_event_connect.h +++ b/src/event/ngx_event_connect.h @@ -25,6 +25,8 @@ typedef ngx_int_t (*ngx_event_get_peer_p void *data); typedef void (*ngx_event_free_peer_pt)(ngx_peer_connection_t *pc, void *data, ngx_uint_t state); +typedef ngx_addr_t *(*ngx_event_get_local_pt)(ngx_peer_connection_t *pc, + void *data); #if (NGX_SSL) typedef ngx_int_t (*ngx_event_set_peer_session_pt)(ngx_peer_connection_t *pc, @@ -45,6 +47,7 @@ struct ngx_peer_connection_s { ngx_event_get_peer_pt get; ngx_event_free_peer_pt free; + ngx_event_get_local_pt local; void *data; #if (NGX_SSL) @@ -56,8 +59,6 @@ struct ngx_peer_connection_s { ngx_atomic_t *lock; #endif - ngx_addr_t *local; - int rcvbuf; ngx_log_t *log; diff --git a/src/http/modules/ngx_http_map_module.c b/src/http/modules/ngx_http_map_module.c --- a/src/http/modules/ngx_http_map_module.c +++ b/src/http/modules/ngx_http_map_module.c @@ -477,7 +477,7 @@ ngx_http_map(ngx_conf_t *cf, ngx_command } var->valid = 1; - var->no_cacheable = 0; + var->no_cacheable = 1; var->not_found = 0; vp = ngx_array_push(&ctx->values_hash[key]); diff --git a/src/http/modules/ngx_http_upstream_keepalive_module.c b/src/http/modules/ngx_http_upstream_keepalive_module.c --- a/src/http/modules/ngx_http_upstream_keepalive_module.c +++ b/src/http/modules/ngx_http_upstream_keepalive_module.c @@ -31,6 +31,7 @@ typedef struct { ngx_event_get_peer_pt original_get_peer; ngx_event_free_peer_pt original_free_peer; + ngx_event_get_local_pt original_get_local; #if (NGX_HTTP_SSL) ngx_event_set_peer_session_pt original_set_session; @@ -63,6 +64,9 @@ static void ngx_http_upstream_keepalive_ static void ngx_http_upstream_keepalive_close_handler(ngx_event_t *ev); static void ngx_http_upstream_keepalive_close(ngx_connection_t *c); +static ngx_addr_t *ngx_http_upstream_get_keepalive_local( + ngx_peer_connection_t *pc, void *data); + #if (NGX_HTTP_SSL) static ngx_int_t ngx_http_upstream_keepalive_set_session( @@ -189,10 +193,12 @@ ngx_http_upstream_init_keepalive_peer(ng kp->data = r->upstream->peer.data; kp->original_get_peer = r->upstream->peer.get; kp->original_free_peer = r->upstream->peer.free; + kp->original_get_local = r->upstream->peer.local; r->upstream->peer.data = kp; r->upstream->peer.get = ngx_http_upstream_get_keepalive_peer; r->upstream->peer.free = ngx_http_upstream_free_keepalive_peer; + r->upstream->peer.local = ngx_http_upstream_get_keepalive_local; #if (NGX_HTTP_SSL) kp->original_set_session = r->upstream->peer.set_session; @@ -430,6 +436,16 @@ ngx_http_upstream_keepalive_close(ngx_co } +static ngx_addr_t * +ngx_http_upstream_get_keepalive_local(ngx_peer_connection_t *pc, + void *data) +{ + ngx_http_upstream_keepalive_peer_data_t *kp = data; + + return kp->original_get_local(pc, kp->data); +} + + #if (NGX_HTTP_SSL) static ngx_int_t diff --git a/src/http/ngx_http_upstream.c b/src/http/ngx_http_upstream.c --- a/src/http/ngx_http_upstream.c +++ b/src/http/ngx_http_upstream.c @@ -141,14 +141,13 @@ static ngx_int_t ngx_http_upstream_respo ngx_http_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_http_upstream_response_length_variable( ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data); +static ngx_int_t ngx_http_upstream_peer_addr_variable(ngx_http_request_t *r, + ngx_http_variable_value_t *v, uintptr_t data); static char *ngx_http_upstream(ngx_conf_t *cf, ngx_command_t *cmd, void *dummy); static char *ngx_http_upstream_server(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); -static ngx_addr_t *ngx_http_upstream_get_local(ngx_http_request_t *r, - ngx_http_upstream_local_t *local); - static void *ngx_http_upstream_create_main_conf(ngx_conf_t *cf); static char *ngx_http_upstream_init_main_conf(ngx_conf_t *cf, void *conf); @@ -366,6 +365,10 @@ static ngx_http_variable_t ngx_http_ups #endif + { ngx_string("upstream_peer_addr"), NULL, + ngx_http_upstream_peer_addr_variable, 0, + NGX_HTTP_VAR_NOCACHEABLE, 0 }, + { ngx_null_string, NULL, NULL, 0, 0, 0 } }; @@ -534,8 +537,6 @@ ngx_http_upstream_init_request(ngx_http_ return; } - u->peer.local = ngx_http_upstream_get_local(r, u->conf->local); - clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module); u->output.alignment = clcf->directio_alignment; @@ -4509,6 +4510,25 @@ ngx_http_upstream_response_length_variab } +static ngx_int_t +ngx_http_upstream_peer_addr_variable(ngx_http_request_t *r, + ngx_http_variable_value_t *v, uintptr_t data) +{ + if (r->upstream == NULL || r->upstream->peer.name == NULL) { + v->not_found = 1; + return NGX_OK; + } + + v->len = r->upstream->peer.name->len; + v->data = r->upstream->peer.name->data; + v->valid = 1; + v->no_cacheable = 0; + v->not_found = 0; + + return NGX_OK; +} + + ngx_int_t ngx_http_upstream_header_variable(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data) @@ -5022,53 +5042,6 @@ ngx_http_upstream_bind_set_slot(ngx_conf } -static ngx_addr_t * -ngx_http_upstream_get_local(ngx_http_request_t *r, - ngx_http_upstream_local_t *local) -{ - ngx_int_t rc; - ngx_str_t val; - ngx_addr_t *addr; - - if (local == NULL) { - return NULL; - } - - if (local->value == NULL) { - return local->addr; - } - - if (ngx_http_complex_value(r, local->value, &val) != NGX_OK) { - return NULL; - } - - if (val.len == 0) { - return NULL; - } - - addr = ngx_palloc(r->pool, sizeof(ngx_addr_t)); - if (addr == NULL) { - return NULL; - } - - rc = ngx_parse_addr(r->pool, addr, val.data, val.len); - - switch (rc) { - case NGX_OK: - addr->name = val; - return addr; - - case NGX_DECLINED: - ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, - "invalid local address \"%V\"", &val); - /* fall through */ - - default: - return NULL; - } -} - - char * ngx_http_upstream_param_set_slot(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) diff --git a/src/http/ngx_http_upstream_round_robin.c b/src/http/ngx_http_upstream_round_robin.c --- a/src/http/ngx_http_upstream_round_robin.c +++ b/src/http/ngx_http_upstream_round_robin.c @@ -22,6 +22,9 @@ static void ngx_http_upstream_empty_save #endif +static ngx_addr_t *ngx_http_upstream_get_local(ngx_peer_connection_t *pc, + void *data); + ngx_int_t ngx_http_upstream_init_round_robin(ngx_conf_t *cf, @@ -246,6 +249,8 @@ ngx_http_upstream_init_round_robin_peer( } } + rrp->request = r; + r->upstream->peer.get = ngx_http_upstream_get_round_robin_peer; r->upstream->peer.free = ngx_http_upstream_free_round_robin_peer; r->upstream->peer.tries = rrp->peers->number; @@ -255,6 +260,7 @@ ngx_http_upstream_init_round_robin_peer( r->upstream->peer.save_session = ngx_http_upstream_save_round_robin_peer_session; #endif + r->upstream->peer.local = ngx_http_upstream_get_local; return NGX_OK; } @@ -679,3 +685,56 @@ ngx_http_upstream_empty_save_session(ngx } #endif + + +static ngx_addr_t * +ngx_http_upstream_get_local(ngx_peer_connection_t *pc, void *data) +{ + ngx_http_upstream_rr_peer_data_t *rrp = data; + + ngx_int_t rc; + ngx_str_t val; + ngx_addr_t *addr; + ngx_http_request_t *r; + ngx_http_upstream_local_t *local; + + r = rrp->request; + local = r->upstream->conf->local; + + if (local == NULL) { + return NULL; + } + + if (local->value == NULL) { + return local->addr; + } + + if (ngx_http_complex_value(r, local->value, &val) != NGX_OK) { + return NULL; + } + + if (val.len == 0) { + return NULL; + } + + addr = ngx_palloc(r->pool, sizeof(ngx_addr_t)); + if (addr == NULL) { + return NULL; + } + + rc = ngx_parse_addr(r->pool, addr, val.data, val.len); + + switch (rc) { + case NGX_OK: + addr->name = val; + return addr; + + case NGX_DECLINED: + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, + "invalid local address \"%V\"", &val); + /* fall through */ + + default: + return NULL; + } +} diff --git a/src/http/ngx_http_upstream_round_robin.h b/src/http/ngx_http_upstream_round_robin.h --- a/src/http/ngx_http_upstream_round_robin.h +++ b/src/http/ngx_http_upstream_round_robin.h @@ -63,6 +63,7 @@ typedef struct { ngx_uint_t current; uintptr_t *tried; uintptr_t data; + ngx_http_request_t *request; } ngx_http_upstream_rr_peer_data_t; From mat999 at gmail.com Mon Dec 9 11:33:29 2013 From: mat999 at gmail.com (SplitIce) Date: Mon, 9 Dec 2013 22:03:29 +1030 Subject: IPv6 & IPv4 backend with proxy_bind In-Reply-To: <20131209102929.GC59059@lo0.su> References: <20131118114554.GB41579@mdounin.ru> <20131118132740.GH41579@mdounin.ru> <20131122132642.GO56821@lo0.su> <20131209102929.GC59059@lo0.su> Message-ID: Im not sure that patch would suit my needs, simply because most of the time I am matching on the upstream address only and need it to work regardless of the port (i.e a regex match on just the IP component). Perhaps your patch could be extended to support - map $upstream_peer_addr $bind_addr { 127.0.0.1:8001 127.0.0.1; 127.0.0.2 127.0.0.1; [::1]:8003 ::1; } Where 127.0.0.2 would match 127.0.0.2:* I certainly like that it doesn't require a conversion to string. I have also attached V2 of my patch (changes to the patch to remove windows path style). On Mon, Dec 9, 2013 at 8:59 PM, Ruslan Ermilov wrote: > On Sat, Nov 23, 2013 at 12:15:28PM +1030, SplitIce wrote: > > Attached is the patch, > > > > This is the first time I have created a variable or really done anything > > inside the http request processing flow so feel free to let me know if > > there is a better way to do something or if I have any edge cases. > > > > This patch provides a $upstream_connecting variable which contains the IP > > address and port of the upstream being connected. If there is no > upstream, > > it will return "-" my understanding is this may happen if the upstream is > > DNS resolved (untested). There may be a better way of doing this? > > > > This should be used in a config like the following - > > map $upstream_connecting $test { > > ~^93\.184\.216\.119\: 192.168.2.40; > > ~^192\.168\.2\.([0-9]+)\: 192.168.2.40; > > } > > > > proxy_bind $test; > > I took a different approach. I've made "local" a peer's method, > so the computation of the local address is delayed until it's > actually needed. By that time, the peer address is already known. > > I've also patched "map" so it creates non-cacheable variables. > > This way, $upstream_peer_addr can be mapped into the local address: > > map $upstream_peer_addr $bind_addr { > 127.0.0.1:8001 127.0.0.1; > 127.0.0.1:8002 127.0.0.1; > [::1]:8003 ::1; > } > > server { > ... > location / { > proxy_pass ...; > proxy_bind $bind_addr; > ... > } > } > > diff --git a/src/event/ngx_event_connect.c b/src/event/ngx_event_connect.c > --- a/src/event/ngx_event_connect.c > +++ b/src/event/ngx_event_connect.c > @@ -17,6 +17,7 @@ ngx_event_connect_peer(ngx_peer_connecti > int rc; > ngx_int_t event; > ngx_err_t err; > + ngx_addr_t *local; > ngx_uint_t level; > ngx_socket_t s; > ngx_event_t *rev, *wev; > @@ -67,9 +68,13 @@ ngx_event_connect_peer(ngx_peer_connecti > } > > if (pc->local) { > - if (bind(s, pc->local->sockaddr, pc->local->socklen) == -1) { > + local = pc->local(pc, pc->data); > + > + if (local != NULL > + && bind(s, local->sockaddr, local->socklen) == -1) > + { > ngx_log_error(NGX_LOG_CRIT, pc->log, ngx_socket_errno, > - "bind(%V) failed", &pc->local->name); > + "bind(%V) failed", &local->name); > > goto failed; > } > diff --git a/src/event/ngx_event_connect.h b/src/event/ngx_event_connect.h > --- a/src/event/ngx_event_connect.h > +++ b/src/event/ngx_event_connect.h > @@ -25,6 +25,8 @@ typedef ngx_int_t (*ngx_event_get_peer_p > void *data); > typedef void (*ngx_event_free_peer_pt)(ngx_peer_connection_t *pc, void > *data, > ngx_uint_t state); > +typedef ngx_addr_t *(*ngx_event_get_local_pt)(ngx_peer_connection_t *pc, > + void *data); > #if (NGX_SSL) > > typedef ngx_int_t (*ngx_event_set_peer_session_pt)(ngx_peer_connection_t > *pc, > @@ -45,6 +47,7 @@ struct ngx_peer_connection_s { > > ngx_event_get_peer_pt get; > ngx_event_free_peer_pt free; > + ngx_event_get_local_pt local; > void *data; > > #if (NGX_SSL) > @@ -56,8 +59,6 @@ struct ngx_peer_connection_s { > ngx_atomic_t *lock; > #endif > > - ngx_addr_t *local; > - > int rcvbuf; > > ngx_log_t *log; > diff --git a/src/http/modules/ngx_http_map_module.c > b/src/http/modules/ngx_http_map_module.c > --- a/src/http/modules/ngx_http_map_module.c > +++ b/src/http/modules/ngx_http_map_module.c > @@ -477,7 +477,7 @@ ngx_http_map(ngx_conf_t *cf, ngx_command > } > > var->valid = 1; > - var->no_cacheable = 0; > + var->no_cacheable = 1; > var->not_found = 0; > > vp = ngx_array_push(&ctx->values_hash[key]); > diff --git a/src/http/modules/ngx_http_upstream_keepalive_module.c > b/src/http/modules/ngx_http_upstream_keepalive_module.c > --- a/src/http/modules/ngx_http_upstream_keepalive_module.c > +++ b/src/http/modules/ngx_http_upstream_keepalive_module.c > @@ -31,6 +31,7 @@ typedef struct { > > ngx_event_get_peer_pt original_get_peer; > ngx_event_free_peer_pt original_free_peer; > + ngx_event_get_local_pt original_get_local; > > #if (NGX_HTTP_SSL) > ngx_event_set_peer_session_pt original_set_session; > @@ -63,6 +64,9 @@ static void ngx_http_upstream_keepalive_ > static void ngx_http_upstream_keepalive_close_handler(ngx_event_t *ev); > static void ngx_http_upstream_keepalive_close(ngx_connection_t *c); > > +static ngx_addr_t *ngx_http_upstream_get_keepalive_local( > + ngx_peer_connection_t *pc, void *data); > + > > #if (NGX_HTTP_SSL) > static ngx_int_t ngx_http_upstream_keepalive_set_session( > @@ -189,10 +193,12 @@ ngx_http_upstream_init_keepalive_peer(ng > kp->data = r->upstream->peer.data; > kp->original_get_peer = r->upstream->peer.get; > kp->original_free_peer = r->upstream->peer.free; > + kp->original_get_local = r->upstream->peer.local; > > r->upstream->peer.data = kp; > r->upstream->peer.get = ngx_http_upstream_get_keepalive_peer; > r->upstream->peer.free = ngx_http_upstream_free_keepalive_peer; > + r->upstream->peer.local = ngx_http_upstream_get_keepalive_local; > > #if (NGX_HTTP_SSL) > kp->original_set_session = r->upstream->peer.set_session; > @@ -430,6 +436,16 @@ ngx_http_upstream_keepalive_close(ngx_co > } > > > +static ngx_addr_t * > +ngx_http_upstream_get_keepalive_local(ngx_peer_connection_t *pc, > + void *data) > +{ > + ngx_http_upstream_keepalive_peer_data_t *kp = data; > + > + return kp->original_get_local(pc, kp->data); > +} > + > + > #if (NGX_HTTP_SSL) > > static ngx_int_t > diff --git a/src/http/ngx_http_upstream.c b/src/http/ngx_http_upstream.c > --- a/src/http/ngx_http_upstream.c > +++ b/src/http/ngx_http_upstream.c > @@ -141,14 +141,13 @@ static ngx_int_t ngx_http_upstream_respo > ngx_http_variable_value_t *v, uintptr_t data); > static ngx_int_t ngx_http_upstream_response_length_variable( > ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data); > +static ngx_int_t ngx_http_upstream_peer_addr_variable(ngx_http_request_t > *r, > + ngx_http_variable_value_t *v, uintptr_t data); > > static char *ngx_http_upstream(ngx_conf_t *cf, ngx_command_t *cmd, void > *dummy); > static char *ngx_http_upstream_server(ngx_conf_t *cf, ngx_command_t *cmd, > void *conf); > > -static ngx_addr_t *ngx_http_upstream_get_local(ngx_http_request_t *r, > - ngx_http_upstream_local_t *local); > - > static void *ngx_http_upstream_create_main_conf(ngx_conf_t *cf); > static char *ngx_http_upstream_init_main_conf(ngx_conf_t *cf, void *conf); > > @@ -366,6 +365,10 @@ static ngx_http_variable_t ngx_http_ups > > #endif > > + { ngx_string("upstream_peer_addr"), NULL, > + ngx_http_upstream_peer_addr_variable, 0, > + NGX_HTTP_VAR_NOCACHEABLE, 0 }, > + > { ngx_null_string, NULL, NULL, 0, 0, 0 } > }; > > @@ -534,8 +537,6 @@ ngx_http_upstream_init_request(ngx_http_ > return; > } > > - u->peer.local = ngx_http_upstream_get_local(r, u->conf->local); > - > clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module); > > u->output.alignment = clcf->directio_alignment; > @@ -4509,6 +4510,25 @@ ngx_http_upstream_response_length_variab > } > > > +static ngx_int_t > +ngx_http_upstream_peer_addr_variable(ngx_http_request_t *r, > + ngx_http_variable_value_t *v, uintptr_t data) > +{ > + if (r->upstream == NULL || r->upstream->peer.name == NULL) { > + v->not_found = 1; > + return NGX_OK; > + } > + > + v->len = r->upstream->peer.name->len; > + v->data = r->upstream->peer.name->data; > + v->valid = 1; > + v->no_cacheable = 0; > + v->not_found = 0; > + > + return NGX_OK; > +} > + > + > ngx_int_t > ngx_http_upstream_header_variable(ngx_http_request_t *r, > ngx_http_variable_value_t *v, uintptr_t data) > @@ -5022,53 +5042,6 @@ ngx_http_upstream_bind_set_slot(ngx_conf > } > > > -static ngx_addr_t * > -ngx_http_upstream_get_local(ngx_http_request_t *r, > - ngx_http_upstream_local_t *local) > -{ > - ngx_int_t rc; > - ngx_str_t val; > - ngx_addr_t *addr; > - > - if (local == NULL) { > - return NULL; > - } > - > - if (local->value == NULL) { > - return local->addr; > - } > - > - if (ngx_http_complex_value(r, local->value, &val) != NGX_OK) { > - return NULL; > - } > - > - if (val.len == 0) { > - return NULL; > - } > - > - addr = ngx_palloc(r->pool, sizeof(ngx_addr_t)); > - if (addr == NULL) { > - return NULL; > - } > - > - rc = ngx_parse_addr(r->pool, addr, val.data, val.len); > - > - switch (rc) { > - case NGX_OK: > - addr->name = val; > - return addr; > - > - case NGX_DECLINED: > - ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, > - "invalid local address \"%V\"", &val); > - /* fall through */ > - > - default: > - return NULL; > - } > -} > - > - > char * > ngx_http_upstream_param_set_slot(ngx_conf_t *cf, ngx_command_t *cmd, > void *conf) > diff --git a/src/http/ngx_http_upstream_round_robin.c > b/src/http/ngx_http_upstream_round_robin.c > --- a/src/http/ngx_http_upstream_round_robin.c > +++ b/src/http/ngx_http_upstream_round_robin.c > @@ -22,6 +22,9 @@ static void ngx_http_upstream_empty_save > > #endif > > +static ngx_addr_t *ngx_http_upstream_get_local(ngx_peer_connection_t *pc, > + void *data); > + > > ngx_int_t > ngx_http_upstream_init_round_robin(ngx_conf_t *cf, > @@ -246,6 +249,8 @@ ngx_http_upstream_init_round_robin_peer( > } > } > > + rrp->request = r; > + > r->upstream->peer.get = ngx_http_upstream_get_round_robin_peer; > r->upstream->peer.free = ngx_http_upstream_free_round_robin_peer; > r->upstream->peer.tries = rrp->peers->number; > @@ -255,6 +260,7 @@ ngx_http_upstream_init_round_robin_peer( > r->upstream->peer.save_session = > > ngx_http_upstream_save_round_robin_peer_session; > #endif > + r->upstream->peer.local = ngx_http_upstream_get_local; > > return NGX_OK; > } > @@ -679,3 +685,56 @@ ngx_http_upstream_empty_save_session(ngx > } > > #endif > + > + > +static ngx_addr_t * > +ngx_http_upstream_get_local(ngx_peer_connection_t *pc, void *data) > +{ > + ngx_http_upstream_rr_peer_data_t *rrp = data; > + > + ngx_int_t rc; > + ngx_str_t val; > + ngx_addr_t *addr; > + ngx_http_request_t *r; > + ngx_http_upstream_local_t *local; > + > + r = rrp->request; > + local = r->upstream->conf->local; > + > + if (local == NULL) { > + return NULL; > + } > + > + if (local->value == NULL) { > + return local->addr; > + } > + > + if (ngx_http_complex_value(r, local->value, &val) != NGX_OK) { > + return NULL; > + } > + > + if (val.len == 0) { > + return NULL; > + } > + > + addr = ngx_palloc(r->pool, sizeof(ngx_addr_t)); > + if (addr == NULL) { > + return NULL; > + } > + > + rc = ngx_parse_addr(r->pool, addr, val.data, val.len); > + > + switch (rc) { > + case NGX_OK: > + addr->name = val; > + return addr; > + > + case NGX_DECLINED: > + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, > + "invalid local address \"%V\"", &val); > + /* fall through */ > + > + default: > + return NULL; > + } > +} > diff --git a/src/http/ngx_http_upstream_round_robin.h > b/src/http/ngx_http_upstream_round_robin.h > --- a/src/http/ngx_http_upstream_round_robin.h > +++ b/src/http/ngx_http_upstream_round_robin.h > @@ -63,6 +63,7 @@ typedef struct { > ngx_uint_t current; > uintptr_t *tried; > uintptr_t data; > + ngx_http_request_t *request; > } ngx_http_upstream_rr_peer_data_t; > > > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: patch-upstream.patch Type: application/octet-stream Size: 3196 bytes Desc: not available URL: From ru at nginx.com Mon Dec 9 12:15:03 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Mon, 9 Dec 2013 16:15:03 +0400 Subject: IPv6 & IPv4 backend with proxy_bind In-Reply-To: References: <20131118114554.GB41579@mdounin.ru> <20131118132740.GH41579@mdounin.ru> <20131122132642.GO56821@lo0.su> <20131209102929.GC59059@lo0.su> Message-ID: <20131209121503.GD59059@lo0.su> On Mon, Dec 09, 2013 at 10:03:29PM +1030, SplitIce wrote: > Im not sure that patch would suit my needs, simply because most of the time > I am matching on the upstream address only and need it to work regardless > of the port (i.e a regex match on just the IP component). > > Perhaps your patch could be extended to support - > > map $upstream_peer_addr $bind_addr { > 127.0.0.1:8001 127.0.0.1; > 127.0.0.2 127.0.0.1; > [::1]:8003 ::1; > } > > Where 127.0.0.2 would match 127.0.0.2:* > > I certainly like that it doesn't require a conversion to string. No need to modify the patch, you can already do it with map: map $upstream_peer_addr $bind_addr { ~^127\.0\.0\. 127.0.0.1; ~^\[::1] ::1; } From steve at stevemorin.com Tue Dec 10 01:14:05 2013 From: steve at stevemorin.com (Steve Morin) Date: Mon, 9 Dec 2013 17:14:05 -0800 Subject: Nginx Logging to Zeromq Module - Sparkngin In-Reply-To: References: Message-ID: Thanks W-Mark [1] I have been looking at that module and a couple others including https://github.com/FRiCKLE/ngx_zeromq [2] My understanding is that nginx uses a number of worker process that are isolated, but need to see how zeromq handles that and if a socket can me shared. Given that they aren't thread safe I assume that is a no. I will pay attention to this part http://zguide.zeromq.org/page:all#Getting-the-Context-Right [3] Saw that, will look where a setup call can happen this is helpful. [4] I have been reading Evan Miller's guide, thanks * Planning on using zero-copy * Not sure if redis would be faster under high load but that would be pretty interesting to see. On Tue, Dec 3, 2013 at 5:49 AM, W-Mark Kubacki wrote: > 2013/11/16 Steve Morin : > > Does anyone have experience integrating zeromq with Nginx. I am looking > for > > some pointers, to see what concerns I should look out for. > > > > I am trying to contribute this code to a open source project. > > -Steve > > This seems to me being a good template for what you want to accomplish: > [1] http://www.binpress.com/app/nginx-redislog-module/998 > > Keep in mind that ??MQ sockets are not threadsafe? and don't share > them between threads. > [2] http://zguide.zeromq.org/page:all#Multithreading-with-MQ > > See also: > [3] http://forum.nginx.org/read.php?2,243942,243986#msg-243986 (master > process ? init_module callback; don't define sockets here; zmq_init > goes here) > [4] http://www.evanmiller.org/nginx-modules-guide.html#definition > > A note on performance: > ? Use zero-copy wherever possible. (I would make a custom string an > optional last part of a message.) ? [5] > http://zeromq.org/blog:zero-copy > ? Does ?MQ have UDP or similar for PUB sockets? If so, use that. > ? I guess that logging or queuing through Redis will perform better > than utilizing ?MQ for that matter iff everything is done on one > machine under high load. > > -- > Mark > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thierry.magnien at sfr.com Tue Dec 10 09:39:39 2013 From: thierry.magnien at sfr.com (MAGNIEN, Thierry) Date: Tue, 10 Dec 2013 09:39:39 +0000 Subject: Nginx Logging to Zeromq Module - Sparkngin In-Reply-To: References: Message-ID: <5D103CE839D50E4CBC62C9FD7B83287C274DD458@EXCN015.encara.local.ads> Hi, >Thanks W-Mark > >[1] I have been looking at that module and a couple others including?https://github.com/FRiCKLE/ngx_zeromq > >[2] My understanding is that nginx uses a number of worker process that are isolated, but need to see how zeromq handles that and if a socket can me shared. ?Given that they aren't thread safe I assume that is a >no. ?I will pay attention to this part?http://zguide.zeromq.org/page:all#Getting-the-Context-Right > >[3] Saw that, will look where a setup call can happen this is helpful. > >[4] ?I have been reading Evan Miller's guide, thanks > >* Planning on using zero-copy >* Not sure if redis would be faster under high load but that would be pretty interesting to see. I don't know much about Zeromq performance, but we pushed redis to about 300k commands per second on our production servers (8 cores), and benchmarks on better hardware (24 cores) show it could handle close to 500k/s. Regards, Thierry From wmark+nginx at hurrikane.de Tue Dec 10 15:00:08 2013 From: wmark+nginx at hurrikane.de (W-Mark Kubacki) Date: Tue, 10 Dec 2013 16:00:08 +0100 Subject: Nginx Logging to Zeromq Module - Sparkngin In-Reply-To: <5D103CE839D50E4CBC62C9FD7B83287C274DD458@EXCN015.encara.local.ads> References: <5D103CE839D50E4CBC62C9FD7B83287C274DD458@EXCN015.encara.local.ads> Message-ID: 2013/12/10 MAGNIEN, Thierry : > > I don't know much about Zeromq performance, but we pushed redis to about 300k commands per second on our production servers (8 cores), and benchmarks on better hardware (24 cores) show it could handle close to 500k/s. ?MQ should come pretty close or even exceed that. You will require a free core for its process, though. Nevertheless implementing it looks rather straightforward (I've done it in a different product) and with an event consumer (Nginx being the producer) you will get one part of a fantastic local analytics tool, similar to Piwik. Here's an introduction to authentication and encryption: [6] http://hintjens.com/blog:49#toc6 Here's how you can prototype the plugin using Lua: [7a] http://wiki.nginx.org/HttpLuaModule#init_by_lua ? initialize context [7b] http://wiki.nginx.org/HttpLuaModule#Data_Sharing_within_an_Nginx_Worker ? the place for PUB sockets [7c] https://github.com/mtourne/nginx_log_by_lua/blob/master/conf/nginx.conf ? collect data for logging, and send a multipart message here? [7d] https://github.com/Neopallium/lua-zmq/blob/master/examples/client_multipart.lua ? ? that's how the call looks like Everything else on the consumer will be counters of some sort. You could still use Redis there for small sites, though. Other noteworthy data structures for the consumer: [8] http://blog.aggregateknowledge.com/2012/10/25/sketch-of-the-day-hyperloglog-cornerstone-of-a-big-data-infrastructure/ [9] http://billmill.org/bloomfilter-tutorial/ [10] http://www.igvita.com/2010/01/06/flow-analysis-time-based-bloom-filters/ I've written this in the hope that other developers of Nginx modules have an idea on where to start (their sockets, for example ;-) ). We should move discussion of further details to a sparkngin group. -- Mark From vbart at nginx.com Tue Dec 10 17:13:32 2013 From: vbart at nginx.com (Valentin Bartenev) Date: Tue, 10 Dec 2013 17:13:32 +0000 Subject: [nginx] SPDY: drop the "delayed" flag when finalizing connection. Message-ID: details: http://hg.nginx.org/nginx/rev/a279d2a33dbf branches: changeset: 5465:a279d2a33dbf user: Valentin Bartenev date: Tue Dec 10 20:27:33 2013 +0400 description: SPDY: drop the "delayed" flag when finalizing connection. This flag in SPDY fake write events serves the same purposes as the "ready" flag in real events, and it must be dropped if request needs to be handled. Otherwise, it can prevent the request from finalization if ngx_http_writer() was set, which results in a connection leak. Found by Xiaochen Wang. diffstat: src/http/ngx_http_spdy.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diffs (13 lines): diff -r 30e806b8636a -r a279d2a33dbf src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Mon Dec 09 10:16:44 2013 +0400 +++ b/src/http/ngx_http_spdy.c Tue Dec 10 20:27:33 2013 +0400 @@ -2855,7 +2855,9 @@ ngx_http_spdy_finalize_connection(ngx_ht if (stream->waiting) { r->blocked -= stream->waiting; stream->waiting = 0; + ev = fc->write; + ev->delayed = 0; } else { ev = fc->read; From agentzh at gmail.com Tue Dec 10 17:14:15 2013 From: agentzh at gmail.com (Yichun Zhang (agentzh)) Date: Tue, 10 Dec 2013 09:14:15 -0800 Subject: [PATCH] Configure: added new option --with-pcre-conf-opt=OPTIONS. Message-ID: Hello! Right now, when PCRE is compiled from source via --with-pcre, there is no way to add custom PCRE configure options for the PCRE build, like --enable-utf8. The following patch adds the new option --with-pcre-conf-opt=OPTIONS for this purpose. Comments welcome as always! :) Thanks! -agentzh # HG changeset patch # User Yichun Zhang # Date 1386694955 28800 # Node ID 9ba6b149669f1f02eeb4cdc0ebd364a949b5c469 # Parent 30e806b8636af5fd3f03ec17df24801f390f7511 Configure: added new option --with-pcre-conf-opt=OPTIONS. diff -r 30e806b8636a -r 9ba6b149669f auto/options --- a/auto/options Mon Dec 09 10:16:44 2013 +0400 +++ b/auto/options Tue Dec 10 09:02:35 2013 -0800 @@ -286,6 +286,7 @@ --with-pcre) USE_PCRE=YES ;; --with-pcre=*) PCRE="$value" ;; --with-pcre-opt=*) PCRE_OPT="$value" ;; + --with-pcre-conf-opt=*) PCRE_CONF_OPT="$value" ;; --with-pcre-jit) PCRE_JIT=YES ;; --with-openssl=*) OPENSSL="$value" ;; @@ -441,6 +442,7 @@ --with-pcre force PCRE library usage --with-pcre=DIR set path to PCRE library sources --with-pcre-opt=OPTIONS set additional build options for PCRE + --with-pcre-conf-opt=OPTIONS set additional configure options for PCRE --with-pcre-jit build PCRE with JIT compilation support --with-md5=DIR set path to md5 library sources -------------- next part -------------- A non-text attachment was scrubbed... Name: pcre-conf-opt.patch Type: text/x-patch Size: 1310 bytes Desc: not available URL: From mdounin at mdounin.ru Tue Dec 10 17:32:03 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 10 Dec 2013 21:32:03 +0400 Subject: [PATCH] Make ngx_http_parse_unsafe_uri() to be able to unescape uri In-Reply-To: <52A3B1EA.3060203@activecloud.com> References: <52A3B1EA.3060203@activecloud.com> Message-ID: <20131210173203.GO95113@mdounin.ru> Hello! On Sun, Dec 08, 2013 at 02:40:26AM +0300, Raman Shishniou wrote: > # HG changeset patch > # User Raman Shishniou > # Date 1386459301 -10800 > # Node ID 54f3670e04e82e00aa424d9773868749026bc693 > # Parent 58716fd3bd2d63c93b0c04fa121232b7126e724b > Make ngx_http_parse_unsafe_uri() to be able to unescape uri > > It makes possible to use an escaped uri with "X-Accel-Redirect" header in > http_upstream module and "Destination" header http_dav module. No need > to unescape uri in http_ssi_filter module any more. > > diff --git a/src/http/modules/ngx_http_dav_module.c b/src/http/modules/ngx_http_dav_module.c > --- a/src/http/modules/ngx_http_dav_module.c > +++ b/src/http/modules/ngx_http_dav_module.c > @@ -517,7 +517,7 @@ > size_t len, root; > ngx_err_t err; > ngx_int_t rc, depth; > - ngx_uint_t overwrite, slash, dir, flags; > + ngx_uint_t overwrite, slash, dir; > ngx_str_t path, uri, duri, args; > ngx_tree_ctx_t tree; > ngx_copy_file_t cf; > @@ -604,9 +604,8 @@ > > duri.len = last - p; > duri.data = p; > - flags = 0; > > - if (ngx_http_parse_unsafe_uri(r, &duri, &args, &flags) != NGX_OK) { > + if (ngx_http_parse_unsafe_uri(r, &duri, &args, 0) != NGX_OK) { > goto invalid_destination; > } Please avoid doing multiple things in the same patch. If you want to change the interface of the ngx_http_parse_unsafe_uri() to accept flags as a value, not as a pointer, please submit a separate patch for this. [...] > diff --git a/src/http/ngx_http_parse.c b/src/http/ngx_http_parse.c > --- a/src/http/ngx_http_parse.c > +++ b/src/http/ngx_http_parse.c > @@ -1778,57 +1778,233 @@ > > ngx_int_t > ngx_http_parse_unsafe_uri(ngx_http_request_t *r, ngx_str_t *uri, > - ngx_str_t *args, ngx_uint_t *flags) > + ngx_str_t *args, ngx_uint_t flags) > { > - u_char ch, *p; > - size_t len; > - > + u_char *src, *dst, *newuri, ch, c, decoded; > + size_t len; > + enum { > + sw_usual = 0, > + sw_quoted, > + sw_quoted_second > + } state; > + > + dst = NULL; > + newuri = NULL; > + src = uri->data; > len = uri->len; > - p = uri->data; > - > - if (len == 0 || p[0] == '?') { > + > + state = 0; > + decoded = 0; > + > + if (len == 0 || src[0] == '?') { > goto unsafe; > } > > - if (p[0] == '.' && len == 3 && p[1] == '.' && (ngx_path_separator(p[2]))) { > - goto unsafe; > - } > - > - for ( /* void */ ; len; len--) { > - > - ch = *p++; > - > - if (usual[ch >> 5] & (1 << (ch & 0x1f))) { > - continue; > - } > - > - if (ch == '?') { > - args->len = len - 1; > - args->data = p; > - uri->len -= len; > - > - return NGX_OK; > - } > - > - if (ch == '\0') { > - goto unsafe; > - } > - > - if (ngx_path_separator(ch) && len > 2) { > - > - /* detect "/../" */ > - > - if (p[0] == '.' && p[1] == '.' && ngx_path_separator(p[2])) { > + while (len--) { > + Changing a for ( /* void */ ; len; len--) { to while (len--) { seems to be something not really needed. > + ch = *src++; > + > + switch (state) { > + case sw_usual: > + if (usual[ch >> 5] & (1 << (ch & 0x1f))) { > + if (dst != NULL) { > + *dst++ = ch; > + } > + break; > + } > + > + if (ch == '\0') { > goto unsafe; > } > + > + if (ch == '?') { > + goto args; > + } > + > + if (ch == '%') { > + if (dst == NULL) { > + newuri = ngx_pnalloc(r->pool, uri->len); > + if (newuri == NULL) { > + return NGX_ERROR; > + } > + if (src - 1 > uri->data) { > + ngx_memcpy(newuri, uri->data, src - uri->data - 1); > + dst = newuri + (src - uri->data - 1); > + } else { > + dst = newuri; > + } > + } > + state = sw_quoted; > + break; > + } > + > + /* detect "../" from begin or "/../" in the middle */ > + if (ngx_path_separator(ch)) { > + if (dst == NULL) { > + > + if (src - 2 == uri->data && > + *(src - 2) == '.' && *(src - 1) == '.') { > + goto unsafe; > + } > + > + if (src - 2 > uri->data && > + ngx_path_separator(*(dst - 3)) && > + *(src - 2) == '.' && *(src - 1) == '.') { > + goto unsafe; > + } > + > + } else { > + > + if (dst - 2 == newuri && > + newuri[0] == '.' && newuri[1] == '.') { > + goto unsafe; > + } > + > + if (dst - 2 > newuri && > + ngx_path_separator(*(dst - 3)) && > + *(dst - 2) == '.' && *(dst - 1) == '.') { > + goto unsafe; > + } > + > + } > + } > + > + if (dst != NULL) { > + *dst++ = ch; > + } > + break; > + > + case sw_quoted: > + if (ch >= '0' && ch <= '9') { > + decoded = (u_char) (ch - '0'); > + state = sw_quoted_second; > + break; > + } > + > + c = (u_char) (ch | 0x20); > + if (c >= 'a' && c <= 'f') { > + decoded = (u_char) (c - 'a' + 10); > + state = sw_quoted_second; > + break; > + } > + > + if (ch == '\0') { > + goto unsafe; > + } > + > + if (ch == '%') { > + *dst++ = '%'; > + break; > + } > + > + if (ch == '?') { > + *dst++ = '%'; > + goto args; > + } > + > + *dst++ = '%'; > + *dst++ = ch; > + state = sw_usual; > + break; > + > + case sw_quoted_second: > + > + state = sw_usual; > + > + if (ch >= '0' && ch <= '9') { > + ch = (u_char) ((decoded << 4) + ch - '0'); > + > + if (ch == '\0') { > + goto unsafe; > + } > + > + *dst++ = ch; > + break; > + } > + > + c = (u_char) (ch | 0x20); > + if (c >= 'a' && c <= 'f') { > + ch = (u_char) ((decoded << 4) + c - 'a' + 10); > + > + /* detect "../" from begin or "/../" in the middle */ > + if (ngx_path_separator(ch)) { > + > + if (dst - 2 == newuri && > + newuri[0] == '.' && newuri[1] == '.') { > + goto unsafe; > + } > + > + if (dst - 2 > newuri && > + ngx_path_separator(*(dst - 3)) && > + *(dst - 2) == '.' && *(dst - 1) == '.') { > + goto unsafe; > + } > + } > + > + *dst++ = ch; > + break; > + } > + > + if (ch == '\0') { > + goto unsafe; > + } > + > + if (ch == '%') { > + *dst++ = '%'; > + *dst++ = *(src - 2); > + state = sw_quoted; > + break; > + } > + > + if (ch == '?') { > + *dst++ = '%'; > + *dst++ = *(src - 2); > + goto args; > + } > + > + *dst++ = '%'; > + *dst++ = *(src - 2); > + *dst++ = ch; > + break; > } > } The code seems to be overcomplicated (and there are lots of minor style problems). It should really be a way to write the code better. E.g., you may consider writing two loops for cases with unmodified uri and new uri allocated. Or, alternatively, one loop to parse simple case without unescaping to find out request arguments and to detect if unescaping is needed, then unescape uri call, and then another loop to check if unescaped uri is still safe. [...] -- Maxim Dounin http://nginx.org/ From steve at stevemorin.com Tue Dec 10 17:56:06 2013 From: steve at stevemorin.com (Steve Morin) Date: Tue, 10 Dec 2013 09:56:06 -0800 Subject: Nginx Logging to Zeromq Module - Sparkngin In-Reply-To: References: <5D103CE839D50E4CBC62C9FD7B83287C274DD458@EXCN015.encara.local.ads> Message-ID: Piotr, You've worked on something similar in the past what's your opinion on the performance of a zeromq module vs redis? -Steve On Tue, Dec 10, 2013 at 7:00 AM, W-Mark Kubacki wrote: > 2013/12/10 MAGNIEN, Thierry : > > > > I don't know much about Zeromq performance, but we pushed redis to about > 300k commands per second on our production servers (8 cores), and > benchmarks on better hardware (24 cores) show it could handle close to > 500k/s. > > ?MQ should come pretty close or even exceed that. You will require a > free core for its process, though. > Nevertheless implementing it looks rather straightforward (I've > done it in a different product) and with an event consumer (Nginx > being the producer) you will get one part of a fantastic local > analytics tool, similar to Piwik. > > Here's an introduction to authentication and encryption: > [6] http://hintjens.com/blog:49#toc6 > > Here's how you can prototype the plugin using Lua: > > [7a] http://wiki.nginx.org/HttpLuaModule#init_by_lua ? initialize context > [7b] > http://wiki.nginx.org/HttpLuaModule#Data_Sharing_within_an_Nginx_Worker > ? the place for PUB sockets > [7c] > https://github.com/mtourne/nginx_log_by_lua/blob/master/conf/nginx.conf > ? collect data for logging, and send a multipart message here? > [7d] > https://github.com/Neopallium/lua-zmq/blob/master/examples/client_multipart.lua > ? ? that's how the call looks like > > Everything else on the consumer will be counters of some sort. You > could still use Redis there for small sites, though. > > Other noteworthy data structures for the consumer: > [8] > http://blog.aggregateknowledge.com/2012/10/25/sketch-of-the-day-hyperloglog-cornerstone-of-a-big-data-infrastructure/ > [9] http://billmill.org/bloomfilter-tutorial/ > [10] > http://www.igvita.com/2010/01/06/flow-analysis-time-based-bloom-filters/ > > I've written this in the hope that other developers of Nginx modules > have an idea on where to start (their sockets, for example ;-) ). We > should move discussion of further details to a sparkngin group. > > -- > Mark > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Tue Dec 10 18:25:28 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 10 Dec 2013 22:25:28 +0400 Subject: [PATCH] Configure: added new option --with-pcre-conf-opt=OPTIONS. In-Reply-To: References: Message-ID: <20131210182528.GQ95113@mdounin.ru> Hello! On Tue, Dec 10, 2013 at 09:14:15AM -0800, Yichun Zhang (agentzh) wrote: > Hello! > > Right now, when PCRE is compiled from source via --with-pcre, there is > no way to add custom PCRE configure options for the PCRE build, like > --enable-utf8. The following patch adds the new option > --with-pcre-conf-opt=OPTIONS for this purpose. > > Comments welcome as always! :) > > Thanks! > -agentzh > > # HG changeset patch > # User Yichun Zhang > # Date 1386694955 28800 > # Node ID 9ba6b149669f1f02eeb4cdc0ebd364a949b5c469 > # Parent 30e806b8636af5fd3f03ec17df24801f390f7511 > Configure: added new option --with-pcre-conf-opt=OPTIONS. > > diff -r 30e806b8636a -r 9ba6b149669f auto/options > --- a/auto/options Mon Dec 09 10:16:44 2013 +0400 > +++ b/auto/options Tue Dec 10 09:02:35 2013 -0800 > @@ -286,6 +286,7 @@ > --with-pcre) USE_PCRE=YES ;; > --with-pcre=*) PCRE="$value" ;; > --with-pcre-opt=*) PCRE_OPT="$value" ;; > + --with-pcre-conf-opt=*) PCRE_CONF_OPT="$value" ;; > --with-pcre-jit) PCRE_JIT=YES ;; > > --with-openssl=*) OPENSSL="$value" ;; > @@ -441,6 +442,7 @@ > --with-pcre force PCRE library usage > --with-pcre=DIR set path to PCRE library sources > --with-pcre-opt=OPTIONS set additional build options for PCRE > + --with-pcre-conf-opt=OPTIONS set additional configure options for PCRE > --with-pcre-jit build PCRE with JIT compilation support I don't like the idea of having both --with-pcre-opt= and --with-pcre-conf-opt= at the same time. Especially if we keep in mind that "--with-...-opt=" means either CFLAGS or configure options depending on the library. (Just in case, an equivalent to passing --enable-utf to PCRE configure would be to use --with-pcre-opt="-DSUPPORT_UTF".) -- Maxim Dounin http://nginx.org/ From rand at sent.com Tue Dec 10 18:31:26 2013 From: rand at sent.com (rand at sent.com) Date: Tue, 10 Dec 2013 10:31:26 -0800 Subject: [PATCH] Configure: added new option --with-pcre-conf-opt=OPTIONS. In-Reply-To: <20131210182528.GQ95113@mdounin.ru> References: <20131210182528.GQ95113@mdounin.ru> Message-ID: <1386700286.26787.57936893.189E53C4@webmail.messagingengine.com> On Tue, Dec 10, 2013, at 10:25 AM, Maxim Dounin wrote: > > Right now, when PCRE is compiled from source via --with-pcre, there is Is the --with-pcre option in nginx, pointing to path to pcre src files, necessarily rebuilding pcre in/with the nginx build? If we've got a locally configured/built/installed pcre instance can nginx simply 'use' that instance? or, is the in-tree build required? Appreciate any clarification. From mdounin at mdounin.ru Tue Dec 10 18:52:31 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 10 Dec 2013 22:52:31 +0400 Subject: [PATCH] Configure: added new option --with-pcre-conf-opt=OPTIONS. In-Reply-To: <1386700286.26787.57936893.189E53C4@webmail.messagingengine.com> References: <20131210182528.GQ95113@mdounin.ru> <1386700286.26787.57936893.189E53C4@webmail.messagingengine.com> Message-ID: <20131210185231.GS95113@mdounin.ru> Hello! On Tue, Dec 10, 2013 at 10:31:26AM -0800, rand at sent.com wrote: > On Tue, Dec 10, 2013, at 10:25 AM, Maxim Dounin wrote: > > > Right now, when PCRE is compiled from source via --with-pcre, there is > > Is the --with-pcre option in nginx, pointing to path to pcre src files, > necessarily rebuilding pcre in/with the nginx build? > > If we've got a locally configured/built/installed pcre instance can > nginx simply 'use' that instance? or, is the in-tree build required? Of course any installed pcre instance can be used. Moreover, nginx will try hard to detect various standard installations, see lib/pcre/conf. The --with-pcre option was introduced to simplify building the library with nginx itself, which is sometimes practical, especially if there is no standard way to install the library (like, e.g., on win32). -- Maxim Dounin http://nginx.org/ From rand at sent.com Tue Dec 10 19:08:03 2013 From: rand at sent.com (rand at sent.com) Date: Tue, 10 Dec 2013 11:08:03 -0800 Subject: [PATCH] Configure: added new option --with-pcre-conf-opt=OPTIONS. In-Reply-To: <20131210185231.GS95113@mdounin.ru> References: <20131210182528.GQ95113@mdounin.ru> <1386700286.26787.57936893.189E53C4@webmail.messagingengine.com> <20131210185231.GS95113@mdounin.ru> Message-ID: <1386702483.8574.57951105.0AF84C3E@webmail.messagingengine.com> > Of course any installed pcre instance can be used. Moreover, > nginx will try hard to detect various standard installations, see > lib/pcre/conf. > > The --with-pcre option was introduced to simplify building the > library with nginx itself, which is sometimes practical, > especially if there is no standard way to install the library > (like, e.g., on win32). reading @ lib/pcre/conf I do note the 'trying hard', although I see no attempts to find in */lib64, or use pkg-config or pcre-config. On my system, src is in /usr/local/src/pcre pcre's configured as shared-only, with pcre-jit enabled, ./configure --enable-shared --disable-static --enable-jit --enable-pcregrep-jit ... and build's installed in /usr/local/bin/pcre-config --version 8.34-RC pcre-config --libs --cflags -L/usr/local/lib64 -lpcre -I/usr/local/include pkg-config libpcre --libs --cflags -I/usr/local/include -L/usr/local/lib64 -lpcre That said, if I want nginx to enable pcre use, use pcre-jit, and use specifically that locally-installed pcre(jit) instance, what's the right nginx config? This, --with-pcre=/usr/local/src/pcre --with-pcre-jit does not appear to be the correct choice. From piotr.sikora at frickle.com Wed Dec 11 00:09:29 2013 From: piotr.sikora at frickle.com (Piotr Sikora) Date: Wed, 11 Dec 2013 01:09:29 +0100 Subject: Nginx Logging to Zeromq Module - Sparkngin In-Reply-To: References: <5D103CE839D50E4CBC62C9FD7B83287C274DD458@EXCN015.encara.local.ads> Message-ID: <073B857F389845CA9115A8EDA1E9F9D8@Desktop> Hey Steve, > You've worked on something similar in the past what's your opinion on the > performance of a zeromq module vs redis? That's rather strange question. ZeroMQ is transport layer, Redis is datastore... That's like asking what's faster: TCP or PostgreSQL? ;) Having said that, if Redis is going to be the final datastore and you're going to use just a single instance of it, then I don't see a reason to add ZeroMQ to the mix. Best regards, Piotr Sikora From rommer at activecloud.com Wed Dec 11 05:48:54 2013 From: rommer at activecloud.com (Raman Shishniou) Date: Wed, 11 Dec 2013 08:48:54 +0300 Subject: [PATCH v2] Make ngx_http_parse_unsafe_uri() to be able to unescape uri In-Reply-To: <20131210173203.GO95113@mdounin.ru> References: <52A3B1EA.3060203@activecloud.com> <20131210173203.GO95113@mdounin.ru> Message-ID: <52A7FCC6.1060503@activecloud.com> # HG changeset patch # User Raman Shishniou # Date 1386740673 -10800 # Node ID 353840924c0ffd1203c8d0894f2128a2548f80c5 # Parent a279d2a33dbfbad511f4415f833c35a60e46bb76 Make ngx_http_parse_unsafe_uri() to be able to unescape uri It makes possible to use an escaped uri with "X-Accel-Redirect" header in http_upstream module and "Destination" header http_dav module. No need to unescape uri in http_ssi_filter module any more. diff --git a/src/http/modules/ngx_http_ssi_filter_module.c b/src/http/modules/ngx_http_ssi_filter_module.c --- a/src/http/modules/ngx_http_ssi_filter_module.c +++ b/src/http/modules/ngx_http_ssi_filter_module.c @@ -1982,8 +1982,6 @@ ngx_http_ssi_include(ngx_http_request_t *r, ngx_http_ssi_ctx_t *ctx, ngx_str_t **params) { - u_char *dst, *src; - size_t len; ngx_int_t rc, key; ngx_str_t *uri, *file, *wait, *set, *stub, args; ngx_buf_t *b; @@ -2054,18 +2052,6 @@ return rc; } - dst = uri->data; - src = uri->data; - - ngx_unescape_uri(&dst, &src, uri->len, NGX_UNESCAPE_URI); - - len = (uri->data + uri->len) - src; - if (len) { - dst = ngx_movemem(dst, src, len); - } - - uri->len = dst - uri->data; - ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "ssi include: \"%V\"", uri); diff --git a/src/http/ngx_http_parse.c b/src/http/ngx_http_parse.c --- a/src/http/ngx_http_parse.c +++ b/src/http/ngx_http_parse.c @@ -9,6 +9,8 @@ #include #include +static ngx_inline ngx_int_t +ngx_http_parse_test_doubledot(const u_char *p, const u_char *begin); static uint32_t usual[] = { 0xffffdbfe, /* 1111 1111 1111 1111 1101 1011 1111 1110 */ @@ -1776,35 +1778,75 @@ } +static ngx_inline ngx_int_t +ngx_http_parse_test_doubledot(const u_char *p, const u_char *begin) +{ + /* assume *p is the path separator or + p points to the next byte after the end */ + + if (p - 2 > begin && + *(p - 1) == '.' && *(p - 2) == '.' && + ngx_path_separator(*(p - 3))) { + return 1; + } + + if (p - 2 == begin && + begin[0] == '.' && begin[1] == '.') { + return 1; + } + + return 0; +} + + ngx_int_t ngx_http_parse_unsafe_uri(ngx_http_request_t *r, ngx_str_t *uri, ngx_str_t *args, ngx_uint_t *flags) { - u_char ch, *p; - size_t len; - + u_char *src, *dst, *newuri, ch, c, decoded; + ngx_int_t unescape; + size_t len; + enum { + sw_usual = 0, + sw_quoted, + sw_quoted_second + } state; + + src = uri->data; len = uri->len; - p = uri->data; - - if (len == 0 || p[0] == '?') { + unescape = 0; + + if (len == 0 || src[0] == '?') { goto unsafe; } - if (p[0] == '.' && len == 3 && p[1] == '.' && (ngx_path_separator(p[2]))) { - goto unsafe; - } - for ( /* void */ ; len; len--) { - ch = *p++; + ch = *src++; if (usual[ch >> 5] & (1 << (ch & 0x1f))) { continue; } + if (ch == '%') { + unescape++; + continue; + } + if (ch == '?') { args->len = len - 1; - args->data = p; + args->data = src; + + if (unescape) { + src--; + goto unescape; + } + + /* detect "/.." at the end or whole uri is ".." */ + if (ngx_http_parse_test_doubledot(src - 1, uri->data)) { + goto unsafe; + } + uri->len -= len; return NGX_OK; @@ -1814,16 +1856,151 @@ goto unsafe; } - if (ngx_path_separator(ch) && len > 2) { - - /* detect "/../" */ - - if (p[0] == '.' && p[1] == '.' && ngx_path_separator(p[2])) { + /* detect "../" at the beginning or "/../" in the middle */ + if (ngx_path_separator(ch) && + ngx_http_parse_test_doubledot(src - 1, uri->data)) { + goto unsafe; + } + } + + /* detect "/.." at the end or whole uri is ".." */ + if (ngx_http_parse_test_doubledot(src, uri->data)) { + goto unsafe; + } + + if (!unescape) { + return NGX_OK; + } + +unescape: + + len = src - uri->data; + newuri = ngx_pnalloc(r->pool, len); + + if (newuri == NULL) { + return NGX_ERROR; + } + + ngx_memcpy(newuri, uri->data, len); + + src = uri->data; + dst = newuri; + decoded = 0; + state = 0; + + for ( /* void */ ; len; len--) { + + ch = *src++; + + switch (state) { + + case sw_usual: + if (usual[ch >> 5] & (1 << (ch & 0x1f))) { + *dst++ = ch; + break; + } + + if (ch == '%') { + state = sw_quoted; + break; + } + + /* detect "../" at the beginning or "/../" in the middle */ + if (ngx_path_separator(ch) && + ngx_http_parse_test_doubledot(dst, newuri)) { goto unsafe; } + + *dst++ = ch; + break; + + case sw_quoted: + if (ch >= '0' && ch <= '9') { + decoded = (u_char) (ch - '0'); + state = sw_quoted_second; + break; + } + + c = (u_char) (ch | 0x20); + if (c >= 'a' && c <= 'f') { + decoded = (u_char) (c - 'a' + 10); + state = sw_quoted_second; + break; + } + + if (ch == '%') { + *dst++ = '%'; + break; + } + + *dst++ = '%'; + *dst++ = ch; + state = sw_usual; + break; + + case sw_quoted_second: + + state = sw_usual; + + if (ch >= '0' && ch <= '9') { + ch = (u_char) ((decoded << 4) + ch - '0'); + + if (ch == '\0') { + goto unsafe; + } + + *dst++ = ch; + break; + } + + c = (u_char) (ch | 0x20); + if (c >= 'a' && c <= 'f') { + ch = (u_char) ((decoded << 4) + c - 'a' + 10); + + /* detect "../" at the beginning or "/../" in the middle */ + if (ngx_path_separator(ch) && + ngx_http_parse_test_doubledot(dst, newuri)) { + goto unsafe; + } + + *dst++ = ch; + break; + } + + if (ch == '%') { + *dst++ = '%'; + *dst++ = *(src - 2); + state = sw_quoted; + break; + } + + *dst++ = '%'; + *dst++ = *(src - 2); + *dst++ = ch; + break; } } + switch (state) { + case sw_usual: + break; + case sw_quoted: + *dst++ = '%'; + break; + case sw_quoted_second: + *dst++ = '%'; + *dst++ = *(src - 1); + break; + } + + /* detect "/.." at the end or whole uri is ".." */ + if (ngx_http_parse_test_doubledot(dst, newuri)) { + goto unsafe; + } + + uri->len = dst - newuri; + uri->data = newuri; + return NGX_OK; unsafe: From rommer at activecloud.com Wed Dec 11 05:58:15 2013 From: rommer at activecloud.com (Raman Shishniou) Date: Wed, 11 Dec 2013 08:58:15 +0300 Subject: [PATCH v2] Make ngx_http_parse_unsafe_uri() to be able to unescape uri In-Reply-To: <52A7FCC6.1060503@activecloud.com> References: <52A3B1EA.3060203@activecloud.com> <20131210173203.GO95113@mdounin.ru> <52A7FCC6.1060503@activecloud.com> Message-ID: <52A7FEF7.8050500@activecloud.com> Hello, This is "simplified" and a little more more secure version of the previous patch. My test program to check the new behavior (linux only): http://pastebin.com/raw.php?i=xDfUyAjw From mdounin at mdounin.ru Wed Dec 11 13:24:54 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 11 Dec 2013 17:24:54 +0400 Subject: [PATCH] Configure: added new option --with-pcre-conf-opt=OPTIONS. In-Reply-To: <1386702483.8574.57951105.0AF84C3E@webmail.messagingengine.com> References: <20131210182528.GQ95113@mdounin.ru> <1386700286.26787.57936893.189E53C4@webmail.messagingengine.com> <20131210185231.GS95113@mdounin.ru> <1386702483.8574.57951105.0AF84C3E@webmail.messagingengine.com> Message-ID: <20131211132454.GT95113@mdounin.ru> Hello! On Tue, Dec 10, 2013 at 11:08:03AM -0800, rand at sent.com wrote: > > Of course any installed pcre instance can be used. Moreover, > > nginx will try hard to detect various standard installations, see > > lib/pcre/conf. > > > > The --with-pcre option was introduced to simplify building the > > library with nginx itself, which is sometimes practical, > > especially if there is no standard way to install the library > > (like, e.g., on win32). > > reading @ > > lib/pcre/conf > > I do note the 'trying hard', although I see no attempts to find in > */lib64, or use pkg-config or pcre-config. Support for pcre-config probably should be added, as it may simplify things. > On my system, src is in > > /usr/local/src/pcre > > pcre's configured as shared-only, with pcre-jit enabled, > > ./configure --enable-shared --disable-static --enable-jit > --enable-pcregrep-jit ... > > and build's installed in > > /usr/local/bin/pcre-config --version > 8.34-RC > pcre-config --libs --cflags > -L/usr/local/lib64 -lpcre > -I/usr/local/include > pkg-config libpcre --libs --cflags > -I/usr/local/include -L/usr/local/lib64 -lpcre > > That said, if I want nginx to enable pcre use, use pcre-jit, and use > specifically that locally-installed pcre(jit) instance, what's the right > nginx config? > > This, > > --with-pcre=/usr/local/src/pcre --with-pcre-jit > > does not appear to be the correct choice. As you already have libpcre installed, you don't need nginx to build it (and you don't need sources). If nginx isn't able to find the pcre itself, use something like this to tell where to look for headers and library: ./configure --with-cc-opt="-I/usr/local/include" \ --with-ld-opt="-L/usr/local/lib64" It should work. -- Maxim Dounin http://nginx.org/ From rand at sent.com Wed Dec 11 15:01:38 2013 From: rand at sent.com (rand at sent.com) Date: Wed, 11 Dec 2013 07:01:38 -0800 Subject: [PATCH] Configure: added new option --with-pcre-conf-opt=OPTIONS. In-Reply-To: <20131211132454.GT95113@mdounin.ru> References: <20131210182528.GQ95113@mdounin.ru> <1386700286.26787.57936893.189E53C4@webmail.messagingengine.com> <20131210185231.GS95113@mdounin.ru> <1386702483.8574.57951105.0AF84C3E@webmail.messagingengine.com> <20131211132454.GT95113@mdounin.ru> Message-ID: <1386774098.11481.58327481.1CFC72B7@webmail.messagingengine.com> > As you already have libpcre installed, you don't need nginx to > build it (and you don't need sources). If nginx isn't able to > find the pcre itself, use something like this to tell where to > look for headers and library: > > ./configure --with-cc-opt="-I/usr/local/include" \ > --with-ld-opt="-L/usr/local/lib64" That's easy enough, then. I've built pcre, separately, with pcre-jit support enabled. Is that sufficient for nginx to use pcre-jit? I suspect that it is, and that "--with-pcrejit" is also simply a build-pcre-in-nginx flag? Thanks From mdounin at mdounin.ru Wed Dec 11 16:08:57 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 11 Dec 2013 20:08:57 +0400 Subject: [PATCH] Configure: added new option --with-pcre-conf-opt=OPTIONS. In-Reply-To: <1386774098.11481.58327481.1CFC72B7@webmail.messagingengine.com> References: <20131210182528.GQ95113@mdounin.ru> <1386700286.26787.57936893.189E53C4@webmail.messagingengine.com> <20131210185231.GS95113@mdounin.ru> <1386702483.8574.57951105.0AF84C3E@webmail.messagingengine.com> <20131211132454.GT95113@mdounin.ru> <1386774098.11481.58327481.1CFC72B7@webmail.messagingengine.com> Message-ID: <20131211160857.GA95113@mdounin.ru> Hello! On Wed, Dec 11, 2013 at 07:01:38AM -0800, rand at sent.com wrote: > > As you already have libpcre installed, you don't need nginx to > > build it (and you don't need sources). If nginx isn't able to > > find the pcre itself, use something like this to tell where to > > look for headers and library: > > > > ./configure --with-cc-opt="-I/usr/local/include" \ > > --with-ld-opt="-L/usr/local/lib64" > > That's easy enough, then. > > I've built pcre, separately, with pcre-jit support enabled. Is that > sufficient for nginx to use pcre-jit? I suspect that it is, and that > "--with-pcrejit" is also simply a build-pcre-in-nginx flag? The "--with-pcre-jit" configure option is documented to mean: $ ./configure --help | grep jit --with-pcre-jit build PCRE with JIT compilation support That is, it's only applies when nginx is going to build PCRE library itself. To actually activate JIT compilation, you'll also need to use pcre_jit directive, see nginx.org/r/pcre_jit. Note that use of PCRE JIT may actually result in lower performance (due to more memory used to compiled regular expressions, and less effective CPU cache as a result), enabling it unconditionally may not be a good idea. -- Maxim Dounin http://nginx.org/ From rand at sent.com Wed Dec 11 18:04:02 2013 From: rand at sent.com (rand at sent.com) Date: Wed, 11 Dec 2013 10:04:02 -0800 Subject: [PATCH] Configure: added new option --with-pcre-conf-opt=OPTIONS. In-Reply-To: <20131211160857.GA95113@mdounin.ru> References: <20131210182528.GQ95113@mdounin.ru> <1386700286.26787.57936893.189E53C4@webmail.messagingengine.com> <20131210185231.GS95113@mdounin.ru> <1386702483.8574.57951105.0AF84C3E@webmail.messagingengine.com> <20131211132454.GT95113@mdounin.ru> <1386774098.11481.58327481.1CFC72B7@webmail.messagingengine.com> <20131211160857.GA95113@mdounin.ru> Message-ID: <1386785042.12234.58409689.68206B7B@webmail.messagingengine.com> > > > ./configure --with-cc-opt="-I/usr/local/include" \ > > > --with-ld-opt="-L/usr/local/lib64" works as promised, ... make ldd objs/nginx | egrep -i pcre libpcre.so.1 => /usr/local/lib64/libpcre.so.1 (0x00007fc62c7e7000) Thanks. > That is, it's only applies when nginx is going to build PCRE library itself. Clear. > To actually activate JIT compilation, you'll also need to use > pcre_jit directive, see nginx.org/r/pcre_jit. > > Note that use of PCRE JIT may actually result in lower performance > (due to more memory used to compiled regular expressions, and less > effective CPU cache as a result), enabling it unconditionally may > not be a good idea. Noted. Thanks! From rand at sent.com Wed Dec 11 18:10:44 2013 From: rand at sent.com (rand at sent.com) Date: Wed, 11 Dec 2013 10:10:44 -0800 Subject: [PATCH] Configure: added new option --with-pcre-conf-opt=OPTIONS. In-Reply-To: <1386785042.12234.58409689.68206B7B@webmail.messagingengine.com> References: <20131210182528.GQ95113@mdounin.ru> <1386700286.26787.57936893.189E53C4@webmail.messagingengine.com> <20131210185231.GS95113@mdounin.ru> <1386702483.8574.57951105.0AF84C3E@webmail.messagingengine.com> <20131211132454.GT95113@mdounin.ru> <1386774098.11481.58327481.1CFC72B7@webmail.messagingengine.com> <20131211160857.GA95113@mdounin.ru> <1386785042.12234.58409689.68206B7B@webmail.messagingengine.com> Message-ID: <1386785444.14396.58411397.0B332471@webmail.messagingengine.com> oops. not quite ... on systems where ld.so.conf does NOT point to the pcre path -- i.e, on my production rather than dev boxes -- the RUNTIME link is incorrect, ldd objs/nginx | egrep -i "pcre" libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007fa719a0d000) This can be remedied by rpath'ing, --with-ld-opt=' ... -L/usr/local/lib64 -Wl,-rpath,/usr/local/lib64 -lpcre' resulting correctly in ldd objs/nginx | egrep -i "pcre" libpcre.so.1 => /usr/local/lib64/libpcre.so.1 (0x00007f961ada1000) *IN*sensitive to local/runtime env From ru at nginx.com Thu Dec 12 11:48:24 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Thu, 12 Dec 2013 15:48:24 +0400 Subject: [PATCH] Configure: added new option --with-pcre-conf-opt=OPTIONS. In-Reply-To: <1386785444.14396.58411397.0B332471@webmail.messagingengine.com> References: <20131210182528.GQ95113@mdounin.ru> <1386700286.26787.57936893.189E53C4@webmail.messagingengine.com> <20131210185231.GS95113@mdounin.ru> <1386702483.8574.57951105.0AF84C3E@webmail.messagingengine.com> <20131211132454.GT95113@mdounin.ru> <1386774098.11481.58327481.1CFC72B7@webmail.messagingengine.com> <20131211160857.GA95113@mdounin.ru> <1386785042.12234.58409689.68206B7B@webmail.messagingengine.com> <1386785444.14396.58411397.0B332471@webmail.messagingengine.com> Message-ID: <20131212114824.GC44032@lo0.su> On Wed, Dec 11, 2013 at 10:10:44AM -0800, rand at sent.com wrote: > oops. not quite ... > > on systems where ld.so.conf does NOT point to the pcre path -- i.e, on > my production rather than dev boxes -- the RUNTIME link is incorrect, > > ldd objs/nginx | egrep -i "pcre" > libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007fa719a0d000) > > This can be remedied by rpath'ing, > > --with-ld-opt=' ... -L/usr/local/lib64 > -Wl,-rpath,/usr/local/lib64 -lpcre' > > resulting correctly in > > ldd objs/nginx | egrep -i "pcre" > libpcre.so.1 => /usr/local/lib64/libpcre.so.1 > (0x00007f961ada1000) > > *IN*sensitive to local/runtime env Setting LD_LIBRARY_PATH in environment might work, please consult with your runtime linker documentation. Some runtime linkers allow you to override paths enforced by -rpath, see LD_LIBRARY_PATH_RPATH in http://man.freebsd.org/rtld for one such example. But this has nothing to do with nginx. From vbart at nginx.com Thu Dec 12 13:58:06 2013 From: vbart at nginx.com (Valentin Bartenev) Date: Thu, 12 Dec 2013 13:58:06 +0000 Subject: [nginx] Use ngx_chain_get_free_buf() in pipe input filters. Message-ID: details: http://hg.nginx.org/nginx/rev/e7b3b9855be8 branches: changeset: 5466:e7b3b9855be8 user: Valentin Bartenev date: Wed Dec 11 21:30:38 2013 +0400 description: Use ngx_chain_get_free_buf() in pipe input filters. No functional changes. diffstat: src/event/ngx_event_pipe.c | 24 +++----------- src/http/modules/ngx_http_fastcgi_module.c | 24 +++----------- src/http/modules/ngx_http_proxy_module.c | 48 ++++++----------------------- 3 files changed, 20 insertions(+), 76 deletions(-) diffs (168 lines): diff -r a279d2a33dbf -r e7b3b9855be8 src/event/ngx_event_pipe.c --- a/src/event/ngx_event_pipe.c Tue Dec 10 20:27:33 2013 +0400 +++ b/src/event/ngx_event_pipe.c Wed Dec 11 21:30:38 2013 +0400 @@ -857,18 +857,12 @@ ngx_event_pipe_copy_input_filter(ngx_eve return NGX_OK; } - if (p->free) { - cl = p->free; - b = cl->buf; - p->free = cl->next; - ngx_free_chain(p->pool, cl); + cl = ngx_chain_get_free_buf(p->pool, &p->free); + if (cl == NULL) { + return NGX_ERROR; + } - } else { - b = ngx_alloc_buf(p->pool); - if (b == NULL) { - return NGX_ERROR; - } - } + b = cl->buf; ngx_memcpy(b, buf, sizeof(ngx_buf_t)); b->shadow = buf; @@ -877,14 +871,6 @@ ngx_event_pipe_copy_input_filter(ngx_eve b->recycled = 1; buf->shadow = b; - cl = ngx_alloc_chain_link(p->pool); - if (cl == NULL) { - return NGX_ERROR; - } - - cl->buf = b; - cl->next = NULL; - ngx_log_debug1(NGX_LOG_DEBUG_EVENT, p->log, 0, "input buf #%d", b->num); if (p->in) { diff -r a279d2a33dbf -r e7b3b9855be8 src/http/modules/ngx_http_fastcgi_module.c --- a/src/http/modules/ngx_http_fastcgi_module.c Tue Dec 10 20:27:33 2013 +0400 +++ b/src/http/modules/ngx_http_fastcgi_module.c Wed Dec 11 21:30:38 2013 +0400 @@ -1827,19 +1827,13 @@ ngx_http_fastcgi_input_filter(ngx_event_ break; } - if (p->free) { - cl = p->free; - b = cl->buf; - p->free = cl->next; - ngx_free_chain(p->pool, cl); - - } else { - b = ngx_alloc_buf(p->pool); - if (b == NULL) { - return NGX_ERROR; - } + cl = ngx_chain_get_free_buf(p->pool, &p->free); + if (cl == NULL) { + return NGX_ERROR; } + b = cl->buf; + ngx_memzero(b, sizeof(ngx_buf_t)); b->pos = f->pos; @@ -1852,14 +1846,6 @@ ngx_http_fastcgi_input_filter(ngx_event_ *prev = b; prev = &b->shadow; - cl = ngx_alloc_chain_link(p->pool); - if (cl == NULL) { - return NGX_ERROR; - } - - cl->buf = b; - cl->next = NULL; - if (p->in) { *p->last_in = cl; } else { diff -r a279d2a33dbf -r e7b3b9855be8 src/http/modules/ngx_http_proxy_module.c --- a/src/http/modules/ngx_http_proxy_module.c Tue Dec 10 20:27:33 2013 +0400 +++ b/src/http/modules/ngx_http_proxy_module.c Wed Dec 11 21:30:38 2013 +0400 @@ -1615,19 +1615,13 @@ ngx_http_proxy_copy_filter(ngx_event_pip return NGX_OK; } - if (p->free) { - cl = p->free; - b = cl->buf; - p->free = cl->next; - ngx_free_chain(p->pool, cl); - - } else { - b = ngx_alloc_buf(p->pool); - if (b == NULL) { - return NGX_ERROR; - } + cl = ngx_chain_get_free_buf(p->pool, &p->free); + if (cl == NULL) { + return NGX_ERROR; } + b = cl->buf; + ngx_memcpy(b, buf, sizeof(ngx_buf_t)); b->shadow = buf; b->tag = p->tag; @@ -1635,14 +1629,6 @@ ngx_http_proxy_copy_filter(ngx_event_pip b->recycled = 1; buf->shadow = b; - cl = ngx_alloc_chain_link(p->pool); - if (cl == NULL) { - return NGX_ERROR; - } - - cl->buf = b; - cl->next = NULL; - ngx_log_debug1(NGX_LOG_DEBUG_EVENT, p->log, 0, "input buf #%d", b->num); if (p->in) { @@ -1707,19 +1693,13 @@ ngx_http_proxy_chunked_filter(ngx_event_ /* a chunk has been parsed successfully */ - if (p->free) { - cl = p->free; - b = cl->buf; - p->free = cl->next; - ngx_free_chain(p->pool, cl); - - } else { - b = ngx_alloc_buf(p->pool); - if (b == NULL) { - return NGX_ERROR; - } + cl = ngx_chain_get_free_buf(p->pool, &p->free); + if (cl == NULL) { + return NGX_ERROR; } + b = cl->buf; + ngx_memzero(b, sizeof(ngx_buf_t)); b->pos = buf->pos; @@ -1732,14 +1712,6 @@ ngx_http_proxy_chunked_filter(ngx_event_ *prev = b; prev = &b->shadow; - cl = ngx_alloc_chain_link(p->pool); - if (cl == NULL) { - return NGX_ERROR; - } - - cl->buf = b; - cl->next = NULL; - if (p->in) { *p->last_in = cl; } else { From rand at sent.com Thu Dec 12 14:25:20 2013 From: rand at sent.com (rand at sent.com) Date: Thu, 12 Dec 2013 06:25:20 -0800 Subject: [PATCH] Configure: added new option --with-pcre-conf-opt=OPTIONS. In-Reply-To: <20131212114824.GC44032@lo0.su> References: <20131210182528.GQ95113@mdounin.ru> <1386700286.26787.57936893.189E53C4@webmail.messagingengine.com> <20131210185231.GS95113@mdounin.ru> <1386702483.8574.57951105.0AF84C3E@webmail.messagingengine.com> <20131211132454.GT95113@mdounin.ru> <1386774098.11481.58327481.1CFC72B7@webmail.messagingengine.com> <20131211160857.GA95113@mdounin.ru> <1386785042.12234.58409689.68206B7B@webmail.messagingengine.com> <1386785444.14396.58411397.0B332471@webmail.messagingengine.com> <20131212114824.GC44032@lo0.su> Message-ID: <1386858320.2620.58771313.7D6E03E7@webmail.messagingengine.com> > Setting LD_LIBRARY_PATH in environment might work, > please consult with your runtime linker documentation. > Some runtime linkers allow you to override paths > enforced by -rpath, see LD_LIBRARY_PATH_RPATH in > http://man.freebsd.org/rtld for one such example. > But this has nothing to do with nginx. env vars are the wrong solution for runtime servers' lib links. rpath is the correct approach -- it's insensitive to environment. if nginx searches for and builds against a given instance of a lib, then the resultant binary should be linked against the specified lib -- consistently & persisently. again, the -Wl,-rpath ldopt spec takes care of it correctly. that IS an nginx issue, inansmuch as it's a choice to correctly rpath specified/found libs, or not. From jefftk at google.com Thu Dec 12 16:29:14 2013 From: jefftk at google.com (Jeff Kaufman) Date: Thu, 12 Dec 2013 11:29:14 -0500 Subject: multiple vary headers and gzip Message-ID: In the ngx_pagespeed module we produce css files which refer to either foo.webp or foo.jpg depending on whether the user agent we're serving them to advertises webp support in the Accept header. We would like to serve these publicly cacheable with "Vary: Accept". Because css is text-based these should also be gzipped when serving to user agents that advertise gzip in the Accept-Encoding header, which means including "Vary: gzip". Currently this means these css files will be served with: Vary: Accept Vary: Accept-Encoding We would rather serve these as: Vary: Accept, Accept-Encoding because many user agents don't properly handle multiple headers with the same name. We're considering adding code to our module like: #if (NGX_HTTP_GZIP) if (r->gzip_vary == 1 && vary_header_already_present(r)) { r->gzip_vary = 0; add_accept_encoding_to_existing_vary_header(r); } #endif This is based on the corresponding code from ngx_http_header_filter_module.cc. This isn't an ideal solution because it couples us more tightly than we'd like to an aspect of nginx internals we're currently relatively separate from, but it's the best we have at the moment. Is there something that would be better for us to do here? (We also considered trying to add a filter after ngx_http_header_filter_module that coalesces duplicate headers with the same name, but it doesn't seem to be possible to modify headers at that stage.) Jeff Kaufman From mdounin at mdounin.ru Thu Dec 12 16:31:26 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 12 Dec 2013 16:31:26 +0000 Subject: [nginx] Trailing whitespace fix. Message-ID: details: http://hg.nginx.org/nginx/rev/f6258a7092e4 branches: changeset: 5467:f6258a7092e4 user: Maxim Dounin date: Thu Dec 12 20:28:48 2013 +0400 description: Trailing whitespace fix. diffstat: src/http/modules/ngx_http_uwsgi_module.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff --git a/src/http/modules/ngx_http_uwsgi_module.c b/src/http/modules/ngx_http_uwsgi_module.c --- a/src/http/modules/ngx_http_uwsgi_module.c +++ b/src/http/modules/ngx_http_uwsgi_module.c @@ -510,7 +510,7 @@ ngx_http_uwsgi_handler(ngx_http_request_ } else { ngx_str_set(&u->schema, "uwsgi://"); - } + } #else ngx_str_set(&u->schema, "uwsgi://"); #endif From aaron.peschel at gmail.com Fri Dec 13 00:30:07 2013 From: aaron.peschel at gmail.com (Aaron Peschel) Date: Thu, 12 Dec 2013 16:30:07 -0800 Subject: Add Support for Variant ETags Message-ID: Hello, Weak ETAGs proved to be insufficient to meet my requirements. Apache has solved their issues with ETAGs by appending the type of the variant to the existing ETAG. For example, the ETAG for a gzip variant would have -gzip appended (EG "abc123" becomes "abc123-gzip"). This seems like a reasonable solution to me, and it should work with Nginx as well. Here is a link to the discussion: https://issues.apache.org/bugzilla/show_bug.cgi?id=39727 I have created a patch to provide support for variant ETAGs in Nginx. Feedback would be appreciated! I would like to see this merged into the main line, and I'm willing to put in any additional work required. Currently I have only switched the gzip module to using variant ETAGs, but others could be switched as well. I have not tested this patch very thoroughly yet either. Thank you, --Aaron Peschel ---- # HG changeset patch # User Aaron Peschel # Date 1386735423 28800 # Tue Dec 10 20:17:03 2013 -0800 # Node ID 1d645b958ca1d086f4ef1d4c0dd63dff6cb2f2af # Parent 1bdf906bba99b4c03d306dd8f2756d4d8ffa4a13 Add Support for Variant ETags Rather than stripping strict ETAGs, we can generate a variant of ETAG on the fly that will act as a strict ETAG for the variant. The solution used by Apache is to simply append the type of the variant to the existing ETAG. This seems like a reasonable solution, and it is what this commit implments. diff -r 1bdf906bba99 -r 1d645b958ca1 src/http/modules/ngx_http_gzip_filter_module.c --- a/src/http/modules/ngx_http_gzip_filter_module.c Fri Dec 06 19:58:27 2013 +0400 +++ b/src/http/modules/ngx_http_gzip_filter_module.c Tue Dec 10 20:17:03 2013 -0800 @@ -306,7 +306,10 @@ ngx_http_clear_content_length(r); ngx_http_clear_accept_ranges(r); - ngx_http_clear_etag(r); + + if (ngx_http_set_etag_variant(r, "gzip") != NGX_OK) { + return NGX_ERROR; + } return ngx_http_next_header_filter(r); } diff -r 1bdf906bba99 -r 1d645b958ca1 src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c Fri Dec 06 19:58:27 2013 +0400 +++ b/src/http/ngx_http_core_module.c Tue Dec 10 20:17:03 2013 -0800 @@ -1855,6 +1855,42 @@ ngx_int_t +ngx_http_set_etag_variant(ngx_http_request_t *r, const char *append_str) +{ + ngx_str_t *etag; + u_char *variant_str; + ngx_int_t append_str_len; + ngx_int_t variant_str_len; + ngx_int_t closing_quote; + + etag = &r->headers_out.etag->value; + append_str_len = ngx_strlen(append_str); + closing_quote = etag->len - 1; + + if (etag->len <= 0) { + return NGX_OK; + } + + // Add one byte for the dash between the etag and the append string. + variant_str = ngx_pnalloc(r->pool, etag->len + append_str_len + 1); + if (variant_str == NULL) { + return NGX_ERROR; + } + + ngx_cpystrn(variant_str, etag->data, closing_quote); + variant_str_len = ngx_sprintf(variant_str + closing_quote, + "-%s\"", + append_str) + - variant_str; + + etag->len = variant_str_len; + etag->data = variant_str; + + return NGX_OK; +} + + +ngx_int_t ngx_http_send_response(ngx_http_request_t *r, ngx_uint_t status, ngx_str_t *ct, ngx_http_complex_value_t *cv) { diff -r 1bdf906bba99 -r 1d645b958ca1 src/http/ngx_http_core_module.h --- a/src/http/ngx_http_core_module.h Fri Dec 06 19:58:27 2013 +0400 +++ b/src/http/ngx_http_core_module.h Tue Dec 10 20:17:03 2013 -0800 @@ -499,6 +499,7 @@ ngx_int_t ngx_http_set_content_type(ngx_http_request_t *r); void ngx_http_set_exten(ngx_http_request_t *r); ngx_int_t ngx_http_set_etag(ngx_http_request_t *r); +ngx_int_t ngx_http_set_etag_variant(ngx_http_request_t *r, const char *append_str); ngx_int_t ngx_http_send_response(ngx_http_request_t *r, ngx_uint_t status, ngx_str_t *ct, ngx_http_complex_value_t *cv); u_char *ngx_http_map_uri_to_path(ngx_http_request_t *r, ngx_str_t *name, -------------- next part -------------- A non-text attachment was scrubbed... Name: etags.patch Type: application/octet-stream Size: 3118 bytes Desc: not available URL: From wangxiaochen0 at gmail.com Fri Dec 13 05:43:20 2013 From: wangxiaochen0 at gmail.com (Xiaochen Wang) Date: Fri, 13 Dec 2013 13:43:20 +0800 Subject: [PATCH] SPDY: fix bug that "tcp_nopush off" doesnt work Message-ID: <20131213054320.GA22200@gmail.com> See the patch as following: # HG changeset patch # User Xiaochen Wang # Date 1386913049 -28800 # Node ID 9af659c4cf71e668183d6a3e2abe1effb89ced22 # Parent f6258a7092e4edc24ee084944eda8a460c549970 SPDY: fix bug that "tcp_nopush off" doesnt work ngx_http_update_location_config() set NGX_TCP_NOPUSH_DISABLED to fake connection originally. In such case, ngx_http_spdy_send_output_queue :: c->send_chain() always sets tcp_nopush for sendfile(). diff -r f6258a7092e4 -r 9af659c4cf71 src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c Thu Dec 12 20:28:48 2013 +0400 +++ b/src/http/ngx_http_core_module.c Fri Dec 13 13:37:29 2013 +0800 @@ -1520,6 +1520,13 @@ if (!clcf->tcp_nopush) { /* disable TCP_NOPUSH/TCP_CORK use */ r->connection->tcp_nopush = NGX_TCP_NOPUSH_DISABLED; + +#if (NGX_HTTP_SPDY) + if (r->spdy_stream) { + r->spdy_stream->connection->connection->tcp_nopush = + NGX_TCP_NOPUSH_DISABLED; + } +#endif } if (r->limit_rate == 0) { -------------- next part -------------- # HG changeset patch # User Xiaochen Wang # Date 1386913049 -28800 # Node ID 9af659c4cf71e668183d6a3e2abe1effb89ced22 # Parent f6258a7092e4edc24ee084944eda8a460c549970 SPDY: fix bug that "tcp_nopush off" doesnt work ngx_http_update_location_config() set NGX_TCP_NOPUSH_DISABLED to fake connection originally. In such case, ngx_http_spdy_send_output_queue :: c->send_chain() always sets tcp_nopush for sendfile(). diff -r f6258a7092e4 -r 9af659c4cf71 src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c Thu Dec 12 20:28:48 2013 +0400 +++ b/src/http/ngx_http_core_module.c Fri Dec 13 13:37:29 2013 +0800 @@ -1520,6 +1520,13 @@ if (!clcf->tcp_nopush) { /* disable TCP_NOPUSH/TCP_CORK use */ r->connection->tcp_nopush = NGX_TCP_NOPUSH_DISABLED; + +#if (NGX_HTTP_SPDY) + if (r->spdy_stream) { + r->spdy_stream->connection->connection->tcp_nopush = + NGX_TCP_NOPUSH_DISABLED; + } +#endif } if (r->limit_rate == 0) { From deymon at inbox.ru Fri Dec 13 07:06:47 2013 From: deymon at inbox.ru (=?UTF-8?B?RGV5bW9u?=) Date: Fri, 13 Dec 2013 11:06:47 +0400 Subject: How to obtain location field in my module In-Reply-To: <20131024163317.GO7074@mdounin.ru> References: <1382626889.825586379@f146.i.mail.ru> <20131024163317.GO7074@mdounin.ru> Message-ID: <1386918407.460614135@f436.i.mail.ru> Hi friends! I have a question: How can I obtain the location path in http-handler which handles requests intended to this location? Here is what I mean: http { ? ... ? server { ??? ... ??? location { # I need value ??????? ... ??? } ??? ... ? } ? ... } ?Thanks for attention! ------------------ Best Regards, Dmitry -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Fri Dec 13 10:45:25 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 13 Dec 2013 14:45:25 +0400 Subject: How to obtain location field in my module In-Reply-To: <1386918407.460614135@f436.i.mail.ru> References: <1382626889.825586379@f146.i.mail.ru> <20131024163317.GO7074@mdounin.ru> <1386918407.460614135@f436.i.mail.ru> Message-ID: <20131213104525.GP95113@mdounin.ru> Hello! On Fri, Dec 13, 2013 at 11:06:47AM +0400, Deymon wrote: > > Hi friends! > I have a question: How can I obtain the location path in http-handler which > handles requests intended to this location? > Here is what I mean: > > http { > ? ... > > ? server { > ??? ... > > ??? location { # I need > value > ??????? ... > ??? } > > ??? ... > ? } > > ? ... > } > > ?Thanks for attention! The is "name" in ngx_http_core_module location config, but note that it's not always a path - there are named and/or regexp locations. E.g., the proxy module does this during configuration merge to later use the location path in internal operations: clcf = ngx_http_conf_get_module_loc_conf(cf, ngx_http_core_module); ... plcf->location = clcf->name; if (clcf->named #if (NGX_PCRE) || clcf->regex #endif || clcf->noname) { if (plcf->vars.uri.len) { ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "\"proxy_pass\" cannot have URI part in " "location given by regular expression, " "or inside named location, " "or inside \"if\" statement, " "or inside \"limit_except\" block"); return NGX_CONF_ERROR; } plcf->location.len = 0; } See http/modules/ngx_http_proxy_module.c for all the code. -- Maxim Dounin http://nginx.org/ From mdounin at mdounin.ru Fri Dec 13 11:12:58 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 13 Dec 2013 15:12:58 +0400 Subject: Add Support for Variant ETags In-Reply-To: References: Message-ID: <20131213111258.GQ95113@mdounin.ru> Hello! On Thu, Dec 12, 2013 at 04:30:07PM -0800, Aaron Peschel wrote: > Hello, > > Weak ETAGs proved to be insufficient to meet my requirements. > > Apache has solved their issues with ETAGs by appending the type of the variant > to the existing ETAG. For example, the ETAG for a gzip variant would have -gzip > appended (EG "abc123" becomes "abc123-gzip"). This seems like a reasonable > solution to me, and it should work with Nginx as well. This is incorrect solution, as gzipping may result in many different byte representations of a resource. Strict entity tags aren't allowed as a result. See also this ticket for some related discussion: http://trac.nginx.org/nginx/ticket/377 -- Maxim Dounin http://nginx.org/ From jefftk at google.com Fri Dec 13 14:59:16 2013 From: jefftk at google.com (Jeff Kaufman) Date: Fri, 13 Dec 2013 09:59:16 -0500 Subject: Add Support for Variant ETags In-Reply-To: <20131213111258.GQ95113@mdounin.ru> References: <20131213111258.GQ95113@mdounin.ru> Message-ID: On Fri, Dec 13, 2013 at 6:12 AM, Maxim Dounin wrote: > gzipping may result in many different byte representations of > a resource. Strict entity tags aren't allowed as a result. > Is gzip deterministic, where you only get different byte representations with different gzip settings? If so we could include the gzip settings (or a hash of them) in the etag. From mdounin at mdounin.ru Fri Dec 13 16:17:44 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 13 Dec 2013 20:17:44 +0400 Subject: Add Support for Variant ETags In-Reply-To: References: <20131213111258.GQ95113@mdounin.ru> Message-ID: <20131213161744.GW95113@mdounin.ru> Hello! On Fri, Dec 13, 2013 at 09:59:16AM -0500, Jeff Kaufman wrote: > On Fri, Dec 13, 2013 at 6:12 AM, Maxim Dounin wrote: > > gzipping may result in many different byte representations of > > a resource. Strict entity tags aren't allowed as a result. > > > > Is gzip deterministic, where you only get different byte > representations with different gzip settings? If so we could include > the gzip settings (or a hash of them) in the etag. No, it's not, at least if flushes are done. As already suggested, I belive the correct solution is to downgrade entity tags to weak ones. -- Maxim Dounin http://nginx.org/ From jefftk at google.com Fri Dec 13 16:29:57 2013 From: jefftk at google.com (Jeff Kaufman) Date: Fri, 13 Dec 2013 11:29:57 -0500 Subject: Add Support for Variant ETags In-Reply-To: <20131213161744.GW95113@mdounin.ru> References: <20131213111258.GQ95113@mdounin.ru> <20131213161744.GW95113@mdounin.ru> Message-ID: On Fri, Dec 13, 2013 at 11:17 AM, Maxim Dounin wrote: > > at least if flushes are done. > Right, thanks! > As already suggested, I believe the correct solution is to > downgrade entity tags to weak ones. > Makes sense. Agreed. From ru at nginx.com Fri Dec 13 16:54:36 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Fri, 13 Dec 2013 16:54:36 +0000 Subject: [nginx] Resolver: improved code readability. Message-ID: details: http://hg.nginx.org/nginx/rev/5c410d6ca7dd branches: changeset: 5468:5c410d6ca7dd user: Ruslan Ermilov date: Fri Dec 06 14:30:27 2013 +0400 description: Resolver: improved code readability. Renamed ngx_resolver_query_t to ngx_resolver_hdr_t as it describes the header that is common to DNS queries and answers. Replaced the magic number 12 by the size of the header structure. The other changes are self-explanatory. diffstat: src/core/ngx_resolver.c | 158 ++++++++++++++++++++++++++--------------------- 1 files changed, 86 insertions(+), 72 deletions(-) diffs (truncated from 401 to 300 lines): diff -r f6258a7092e4 -r 5c410d6ca7dd src/core/ngx_resolver.c --- a/src/core/ngx_resolver.c Thu Dec 12 20:28:48 2013 +0400 +++ b/src/core/ngx_resolver.c Fri Dec 06 14:30:27 2013 +0400 @@ -26,7 +26,7 @@ typedef struct { u_char nns_lo; u_char nar_hi; u_char nar_lo; -} ngx_resolver_query_t; +} ngx_resolver_hdr_t; typedef struct { @@ -1021,31 +1021,31 @@ ngx_resolver_read_response(ngx_event_t * static void ngx_resolver_process_response(ngx_resolver_t *r, u_char *buf, size_t n) { - char *err; - size_t len; - ngx_uint_t i, times, ident, qident, flags, code, nqs, nan, - qtype, qclass; - ngx_queue_t *q; - ngx_resolver_qs_t *qs; - ngx_resolver_node_t *rn; - ngx_resolver_query_t *query; - - if ((size_t) n < sizeof(ngx_resolver_query_t)) { + char *err; + size_t len; + ngx_uint_t i, times, ident, qident, flags, code, nqs, nan, + qtype, qclass; + ngx_queue_t *q; + ngx_resolver_qs_t *qs; + ngx_resolver_hdr_t *response; + ngx_resolver_node_t *rn; + + if (n < sizeof(ngx_resolver_hdr_t)) { goto short_response; } - query = (ngx_resolver_query_t *) buf; - - ident = (query->ident_hi << 8) + query->ident_lo; - flags = (query->flags_hi << 8) + query->flags_lo; - nqs = (query->nqs_hi << 8) + query->nqs_lo; - nan = (query->nan_hi << 8) + query->nan_lo; + response = (ngx_resolver_hdr_t *) buf; + + ident = (response->ident_hi << 8) + response->ident_lo; + flags = (response->flags_hi << 8) + response->flags_lo; + nqs = (response->nqs_hi << 8) + response->nqs_lo; + nan = (response->nan_hi << 8) + response->nan_lo; ngx_log_debug6(NGX_LOG_DEBUG_CORE, r->log, 0, "resolver DNS response %ui fl:%04Xui %ui/%ui/%ud/%ud", ident, flags, nqs, nan, - (query->nns_hi << 8) + query->nns_lo, - (query->nar_hi << 8) + query->nar_lo); + (response->nns_hi << 8) + response->nns_lo, + (response->nar_hi << 8) + response->nar_lo); if (!(flags & 0x8000)) { ngx_log_error(r->log_level, r->log, 0, @@ -1087,7 +1087,7 @@ ngx_resolver_process_response(ngx_resolv goto done; } - i = sizeof(ngx_resolver_query_t); + i = sizeof(ngx_resolver_hdr_t); while (i < (ngx_uint_t) n) { if (buf[i] == '\0') { @@ -1152,7 +1152,7 @@ found: short_response: - err = "short dns response"; + err = "short DNS response"; done: @@ -1180,12 +1180,15 @@ ngx_resolver_process_a(ngx_resolver_t *r uint32_t hash; in_addr_t addr, *addrs; ngx_str_t name; - ngx_uint_t qtype, qident, naddrs, a, i, n, start; + ngx_uint_t type, qident, naddrs, a, i, n, start; ngx_resolver_an_t *an; ngx_resolver_ctx_t *ctx, *next; ngx_resolver_node_t *rn; - if (ngx_resolver_copy(r, &name, buf, &buf[12], &buf[last]) != NGX_OK) { + if (ngx_resolver_copy(r, &name, buf, + buf + sizeof(ngx_resolver_hdr_t), buf + last) + != NGX_OK) + { return; } @@ -1215,7 +1218,7 @@ ngx_resolver_process_a(ngx_resolver_t *r ngx_resolver_free(r, name.data); if (code == 0 && nan == 0) { - code = 3; /* NXDOMAIN */ + code = NGX_RESOLVE_NXDOMAIN; } if (code) { @@ -1246,7 +1249,6 @@ ngx_resolver_process_a(ngx_resolver_t *r addr = 0; addrs = NULL; cname = NULL; - qtype = 0; ttl = 0; for (a = 0; a < nan; a++) { @@ -1273,7 +1275,7 @@ ngx_resolver_process_a(ngx_resolver_t *r test_length: if (i - start < 2) { - err = "invalid name in dns response"; + err = "invalid name in DNS response"; goto invalid; } @@ -1285,7 +1287,7 @@ ngx_resolver_process_a(ngx_resolver_t *r an = (ngx_resolver_an_t *) &buf[i]; - qtype = (an->type_hi << 8) + an->type_lo; + type = (an->type_hi << 8) + an->type_lo; len = (an->len_hi << 8) + an->len_lo; ttl = (an->ttl[0] << 24) + (an->ttl[1] << 16) + (an->ttl[2] << 8) + (an->ttl[3]); @@ -1294,7 +1296,9 @@ ngx_resolver_process_a(ngx_resolver_t *r ttl = 0; } - if (qtype == NGX_RESOLVE_A) { + switch (type) { + + case NGX_RESOLVE_A: i += sizeof(ngx_resolver_an_t); @@ -1309,16 +1313,25 @@ ngx_resolver_process_a(ngx_resolver_t *r i += len; - } else if (qtype == NGX_RESOLVE_CNAME) { + break; + + case NGX_RESOLVE_CNAME: + cname = &buf[i] + sizeof(ngx_resolver_an_t); i += sizeof(ngx_resolver_an_t) + len; - } else if (qtype == NGX_RESOLVE_DNAME) { + break; + + case NGX_RESOLVE_DNAME: + i += sizeof(ngx_resolver_an_t) + len; - } else { + break; + + default: + ngx_log_error(r->log_level, r->log, 0, - "unexpected qtype %ui", qtype); + "unexpected RR type %ui", type); } } @@ -1347,27 +1360,25 @@ ngx_resolver_process_a(ngx_resolver_t *r if (buf[i] & 0xc0) { i += 2; - goto ok; + break; } if (buf[i] == 0) { i++; - goto ok; + break; } i += 1 + buf[i]; } - ok: - an = (ngx_resolver_an_t *) &buf[i]; - qtype = (an->type_hi << 8) + an->type_lo; + type = (an->type_hi << 8) + an->type_lo; len = (an->len_hi << 8) + an->len_lo; i += sizeof(ngx_resolver_an_t); - if (qtype == NGX_RESOLVE_A) { + if (type == NGX_RESOLVE_A) { addrs[n++] = htonl((buf[i] << 24) + (buf[i + 1] << 16) + (buf[i + 2] << 8) + (buf[i + 3])); @@ -1422,12 +1433,13 @@ ngx_resolver_process_a(ngx_resolver_t *r rn->query = NULL; return; - - } else if (cname) { + } + + if (cname) { /* CNAME only */ - if (ngx_resolver_copy(r, &name, buf, cname, &buf[last]) != NGX_OK) { + if (ngx_resolver_copy(r, &name, buf, cname, buf + last) != NGX_OK) { return; } @@ -1460,13 +1472,12 @@ ngx_resolver_process_a(ngx_resolver_t *r } ngx_log_error(r->log_level, r->log, 0, - "no A or CNAME types in DNS responses, unknown query type: %ui", - qtype); + "no A or CNAME types in DNS response"); return; short_response: - err = "short dns response"; + err = "short DNS response"; invalid: @@ -1494,29 +1505,32 @@ ngx_resolver_process_ptr(ngx_resolver_t size_t len; in_addr_t addr; int32_t ttl; - ngx_int_t digit; + ngx_int_t octet; ngx_str_t name; ngx_uint_t i, mask, qident; ngx_resolver_an_t *an; ngx_resolver_ctx_t *ctx, *next; ngx_resolver_node_t *rn; - if (ngx_resolver_copy(r, NULL, buf, &buf[12], &buf[n]) != NGX_OK) { + if (ngx_resolver_copy(r, NULL, buf, + buf + sizeof(ngx_resolver_hdr_t), buf + n) + != NGX_OK) + { goto invalid_in_addr_arpa; } addr = 0; - i = 12; + i = sizeof(ngx_resolver_hdr_t); for (mask = 0; mask < 32; mask += 8) { len = buf[i++]; - digit = ngx_atoi(&buf[i], len); - if (digit == NGX_ERROR || digit > 255) { + octet = ngx_atoi(&buf[i], len); + if (octet == NGX_ERROR || octet > 255) { goto invalid_in_addr_arpa; } - addr += digit << mask; + addr += octet << mask; i += len; } @@ -1547,7 +1561,7 @@ ngx_resolver_process_ptr(ngx_resolver_t } if (code == 0 && nan == 0) { - code = 3; /* NXDOMAIN */ + code = NGX_RESOLVE_NXDOMAIN; } if (code) { @@ -1581,7 +1595,7 @@ ngx_resolver_process_ptr(ngx_resolver_t /* compression pointer to "XX.XX.XX.XX.in-addr.arpa */ - if (buf[i] != 0xc0 || buf[i + 1] != 0x0c) { + if (buf[i] != 0xc0 || buf[i + 1] != sizeof(ngx_resolver_hdr_t)) { err = "invalid in-addr.arpa name in DNS response"; goto invalid; } @@ -1607,7 +1621,7 @@ ngx_resolver_process_ptr(ngx_resolver_t goto short_response; } From ru at nginx.com Fri Dec 13 16:54:37 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Fri, 13 Dec 2013 16:54:37 +0000 Subject: [nginx] Resolver: removed unnecessary casts. Message-ID: details: http://hg.nginx.org/nginx/rev/b2fc466a11a7 branches: changeset: 5469:b2fc466a11a7 user: Ruslan Ermilov date: Fri Dec 06 14:30:27 2013 +0400 description: Resolver: removed unnecessary casts. diffstat: src/core/ngx_resolver.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (21 lines): diff -r 5c410d6ca7dd -r b2fc466a11a7 src/core/ngx_resolver.c --- a/src/core/ngx_resolver.c Fri Dec 06 14:30:27 2013 +0400 +++ b/src/core/ngx_resolver.c Fri Dec 06 14:30:27 2013 +0400 @@ -1589,7 +1589,7 @@ ngx_resolver_process_ptr(ngx_resolver_t i += sizeof("\7in-addr\4arpa") + sizeof(ngx_resolver_qs_t); - if (i + 2 + sizeof(ngx_resolver_an_t) > (ngx_uint_t) n) { + if (i + 2 + sizeof(ngx_resolver_an_t) > n) { goto short_response; } @@ -1617,7 +1617,7 @@ ngx_resolver_process_ptr(ngx_resolver_t i += 2 + sizeof(ngx_resolver_an_t); - if (i + len > (ngx_uint_t) n) { + if (i + len > n) { goto short_response; } From ru at nginx.com Fri Dec 13 16:54:39 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Fri, 13 Dec 2013 16:54:39 +0000 Subject: [nginx] Resolver: fixed response processing. Message-ID: details: http://hg.nginx.org/nginx/rev/aebdca7e8f8f branches: changeset: 5470:aebdca7e8f8f user: Ruslan Ermilov date: Fri Dec 06 14:30:27 2013 +0400 description: Resolver: fixed response processing. Stricten response header checks: ensure that reserved bits are zeroes, and that the opcode is "standard query". Fixed the "zero-length domain name in DNS response" condition. diffstat: src/core/ngx_resolver.c | 11 +++++------ 1 files changed, 5 insertions(+), 6 deletions(-) diffs (46 lines): diff -r b2fc466a11a7 -r aebdca7e8f8f src/core/ngx_resolver.c --- a/src/core/ngx_resolver.c Fri Dec 06 14:30:27 2013 +0400 +++ b/src/core/ngx_resolver.c Fri Dec 06 14:30:27 2013 +0400 @@ -1022,7 +1022,6 @@ static void ngx_resolver_process_response(ngx_resolver_t *r, u_char *buf, size_t n) { char *err; - size_t len; ngx_uint_t i, times, ident, qident, flags, code, nqs, nan, qtype, qclass; ngx_queue_t *q; @@ -1047,13 +1046,14 @@ ngx_resolver_process_response(ngx_resolv (response->nns_hi << 8) + response->nns_lo, (response->nar_hi << 8) + response->nar_lo); - if (!(flags & 0x8000)) { + /* response to a standard query */ + if ((flags & 0xf870) != 0x8000) { ngx_log_error(r->log_level, r->log, 0, "invalid DNS response %ui fl:%04Xui", ident, flags); return; } - code = flags & 0x7f; + code = flags & 0xf; if (code == NGX_RESOLVE_FORMERR) { @@ -1094,15 +1094,14 @@ ngx_resolver_process_response(ngx_resolv goto found; } - len = buf[i]; - i += 1 + len; + i += 1 + buf[i]; } goto short_response; found: - if (i++ == 0) { + if (i++ == sizeof(ngx_resolver_hdr_t)) { err = "zero-length domain name in DNS response"; goto done; } From ru at nginx.com Fri Dec 13 16:54:40 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Fri, 13 Dec 2013 16:54:40 +0000 Subject: [nginx] Resolver: fixes in A processing. Message-ID: details: http://hg.nginx.org/nginx/rev/9c96782d9d05 branches: changeset: 5471:9c96782d9d05 user: Ruslan Ermilov date: Fri Dec 06 14:30:27 2013 +0400 description: Resolver: fixes in A processing. Verify that class of RR is "IN". Verify that RR data length is exactly four octets. Correctly shift to the next RR if RR type is unknown. diffstat: src/core/ngx_resolver.c | 41 ++++++++++++++++++++++++++--------------- 1 files changed, 26 insertions(+), 15 deletions(-) diffs (142 lines): diff -r aebdca7e8f8f -r 9c96782d9d05 src/core/ngx_resolver.c --- a/src/core/ngx_resolver.c Fri Dec 06 14:30:27 2013 +0400 +++ b/src/core/ngx_resolver.c Fri Dec 06 14:30:27 2013 +0400 @@ -1179,7 +1179,7 @@ ngx_resolver_process_a(ngx_resolver_t *r uint32_t hash; in_addr_t addr, *addrs; ngx_str_t name; - ngx_uint_t type, qident, naddrs, a, i, n, start; + ngx_uint_t type, class, qident, naddrs, a, i, n, start; ngx_resolver_an_t *an; ngx_resolver_ctx_t *ctx, *next; ngx_resolver_node_t *rn; @@ -1202,6 +1202,7 @@ ngx_resolver_process_a(ngx_resolver_t *r if (rn == NULL || rn->query == NULL) { ngx_log_error(r->log_level, r->log, 0, "unexpected response for %V", &name); + ngx_resolver_free(r, name.data); goto failed; } @@ -1211,6 +1212,7 @@ ngx_resolver_process_a(ngx_resolver_t *r ngx_log_error(r->log_level, r->log, 0, "wrong ident %ui response for %V, expect %ui", ident, &name, qident); + ngx_resolver_free(r, name.data); goto failed; } @@ -1287,21 +1289,33 @@ ngx_resolver_process_a(ngx_resolver_t *r an = (ngx_resolver_an_t *) &buf[i]; type = (an->type_hi << 8) + an->type_lo; + class = (an->class_hi << 8) + an->class_lo; len = (an->len_hi << 8) + an->len_lo; ttl = (an->ttl[0] << 24) + (an->ttl[1] << 16) + (an->ttl[2] << 8) + (an->ttl[3]); + if (class != 1) { + ngx_log_error(r->log_level, r->log, 0, + "unexpected RR class %ui", class); + goto failed; + } + if (ttl < 0) { ttl = 0; } + i += sizeof(ngx_resolver_an_t); + switch (type) { case NGX_RESOLVE_A: - i += sizeof(ngx_resolver_an_t); - - if (i + len > last) { + if (len != 4) { + err = "invalid A record in DNS response"; + goto invalid; + } + + if (i + 4 > last) { goto short_response; } @@ -1310,21 +1324,16 @@ ngx_resolver_process_a(ngx_resolver_t *r naddrs++; - i += len; - break; case NGX_RESOLVE_CNAME: - cname = &buf[i] + sizeof(ngx_resolver_an_t); - i += sizeof(ngx_resolver_an_t) + len; + cname = &buf[i]; break; case NGX_RESOLVE_DNAME: - i += sizeof(ngx_resolver_an_t) + len; - break; default: @@ -1332,6 +1341,8 @@ ngx_resolver_process_a(ngx_resolver_t *r ngx_log_error(r->log_level, r->log, 0, "unexpected RR type %ui", type); } + + i += len; } ngx_log_debug3(NGX_LOG_DEBUG_CORE, r->log, 0, @@ -1347,7 +1358,7 @@ ngx_resolver_process_a(ngx_resolver_t *r addrs = ngx_resolver_alloc(r, naddrs * sizeof(in_addr_t)); if (addrs == NULL) { - return; + goto failed; } n = 0; @@ -1395,7 +1406,7 @@ ngx_resolver_process_a(ngx_resolver_t *r addrs = ngx_resolver_dup(r, rn->u.addrs, naddrs * sizeof(in_addr_t)); if (addrs == NULL) { - return; + goto failed; } } @@ -1439,7 +1450,7 @@ ngx_resolver_process_a(ngx_resolver_t *r /* CNAME only */ if (ngx_resolver_copy(r, &name, buf, cname, buf + last) != NGX_OK) { - return; + goto failed; } ngx_log_debug1(NGX_LOG_DEBUG_CORE, r->log, 0, @@ -1467,6 +1478,8 @@ ngx_resolver_process_a(ngx_resolver_t *r ngx_resolver_free(r, rn->query); rn->query = NULL; + /* unlock name mutex */ + return; } @@ -1490,8 +1503,6 @@ failed: /* unlock name mutex */ - ngx_resolver_free(r, name.data); - return; } From ru at nginx.com Fri Dec 13 16:54:42 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Fri, 13 Dec 2013 16:54:42 +0000 Subject: [nginx] Resolver: fixes in PTR processing. Message-ID: details: http://hg.nginx.org/nginx/rev/ab493c60d9ff branches: changeset: 5472:ab493c60d9ff user: Ruslan Ermilov date: Fri Dec 06 14:30:27 2013 +0400 description: Resolver: fixes in PTR processing. Verify that class of RR is "IN". Verify that RR data length is non-zero. diffstat: src/core/ngx_resolver.c | 17 ++++++++++++----- 1 files changed, 12 insertions(+), 5 deletions(-) diffs (66 lines): diff -r 9c96782d9d05 -r ab493c60d9ff src/core/ngx_resolver.c --- a/src/core/ngx_resolver.c Fri Dec 06 14:30:27 2013 +0400 +++ b/src/core/ngx_resolver.c Fri Dec 06 14:30:27 2013 +0400 @@ -1517,7 +1517,7 @@ ngx_resolver_process_ptr(ngx_resolver_t int32_t ttl; ngx_int_t octet; ngx_str_t name; - ngx_uint_t i, mask, qident; + ngx_uint_t i, mask, qident, class; ngx_resolver_an_t *an; ngx_resolver_ctx_t *ctx, *next; ngx_resolver_node_t *rn; @@ -1526,7 +1526,7 @@ ngx_resolver_process_ptr(ngx_resolver_t buf + sizeof(ngx_resolver_hdr_t), buf + n) != NGX_OK) { - goto invalid_in_addr_arpa; + return; } addr = 0; @@ -1599,7 +1599,7 @@ ngx_resolver_process_ptr(ngx_resolver_t i += sizeof("\7in-addr\4arpa") + sizeof(ngx_resolver_qs_t); - if (i + 2 + sizeof(ngx_resolver_an_t) > n) { + if (i + 2 + sizeof(ngx_resolver_an_t) >= n) { goto short_response; } @@ -1612,10 +1612,17 @@ ngx_resolver_process_ptr(ngx_resolver_t an = (ngx_resolver_an_t *) &buf[i + 2]; + class = (an->class_hi << 8) + an->class_lo; len = (an->len_hi << 8) + an->len_lo; ttl = (an->ttl[0] << 24) + (an->ttl[1] << 16) + (an->ttl[2] << 8) + (an->ttl[3]); + if (class != 1) { + ngx_log_error(r->log_level, r->log, 0, + "unexpected RR class %ui", class); + goto failed; + } + if (ttl < 0) { ttl = 0; } @@ -1623,7 +1630,7 @@ ngx_resolver_process_ptr(ngx_resolver_t ngx_log_debug3(NGX_LOG_DEBUG_CORE, r->log, 0, "resolver qt:%ui cl:%ui len:%uz", (an->type_hi << 8) + an->type_lo, - (an->class_hi << 8) + an->class_lo, len); + class, len); i += 2 + sizeof(ngx_resolver_an_t); @@ -1632,7 +1639,7 @@ ngx_resolver_process_ptr(ngx_resolver_t } if (ngx_resolver_copy(r, &name, buf, buf + i, buf + n) != NGX_OK) { - return; + goto failed; } ngx_log_debug1(NGX_LOG_DEBUG_CORE, r->log, 0, "resolver an:%V", &name); From ru at nginx.com Fri Dec 13 16:54:43 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Fri, 13 Dec 2013 16:54:43 +0000 Subject: [nginx] Core: externalized ngx_cmp_sockaddr(). Message-ID: details: http://hg.nginx.org/nginx/rev/d39ef821d03e branches: changeset: 5473:d39ef821d03e user: Ruslan Ermilov date: Fri Dec 06 14:30:27 2013 +0400 description: Core: externalized ngx_cmp_sockaddr(). It's also extended with the "cmp_port" argument to indicate whether ports should be compared as well, or only addresses. diffstat: src/core/ngx_cycle.c | 73 ++------------------------------------------------- src/core/ngx_inet.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++ src/core/ngx_inet.h | 2 + 3 files changed, 78 insertions(+), 70 deletions(-) diffs (189 lines): diff -r ab493c60d9ff -r d39ef821d03e src/core/ngx_cycle.c --- a/src/core/ngx_cycle.c Fri Dec 06 14:30:27 2013 +0400 +++ b/src/core/ngx_cycle.c Fri Dec 06 14:30:27 2013 +0400 @@ -11,7 +11,6 @@ static void ngx_destroy_cycle_pools(ngx_conf_t *conf); -static ngx_int_t ngx_cmp_sockaddr(struct sockaddr *sa1, struct sockaddr *sa2); static ngx_int_t ngx_init_zone_pool(ngx_cycle_t *cycle, ngx_shm_zone_t *shm_zone); static ngx_int_t ngx_test_lockfile(u_char *file, ngx_log_t *log); @@ -494,7 +493,9 @@ ngx_init_cycle(ngx_cycle_t *old_cycle) continue; } - if (ngx_cmp_sockaddr(nls[n].sockaddr, ls[i].sockaddr) == NGX_OK) + if (ngx_cmp_sockaddr(nls[n].sockaddr, nls[n].socklen, + ls[i].sockaddr, ls[n].socklen, 1) + == NGX_OK) { nls[n].fd = ls[i].fd; nls[n].previous = &ls[i]; @@ -849,74 +850,6 @@ ngx_destroy_cycle_pools(ngx_conf_t *conf static ngx_int_t -ngx_cmp_sockaddr(struct sockaddr *sa1, struct sockaddr *sa2) -{ - struct sockaddr_in *sin1, *sin2; -#if (NGX_HAVE_INET6) - struct sockaddr_in6 *sin61, *sin62; -#endif -#if (NGX_HAVE_UNIX_DOMAIN) - struct sockaddr_un *saun1, *saun2; -#endif - - if (sa1->sa_family != sa2->sa_family) { - return NGX_DECLINED; - } - - switch (sa1->sa_family) { - -#if (NGX_HAVE_INET6) - case AF_INET6: - sin61 = (struct sockaddr_in6 *) sa1; - sin62 = (struct sockaddr_in6 *) sa2; - - if (sin61->sin6_port != sin62->sin6_port) { - return NGX_DECLINED; - } - - if (ngx_memcmp(&sin61->sin6_addr, &sin62->sin6_addr, 16) != 0) { - return NGX_DECLINED; - } - - break; -#endif - -#if (NGX_HAVE_UNIX_DOMAIN) - case AF_UNIX: - saun1 = (struct sockaddr_un *) sa1; - saun2 = (struct sockaddr_un *) sa2; - - if (ngx_memcmp(&saun1->sun_path, &saun2->sun_path, - sizeof(saun1->sun_path)) - != 0) - { - return NGX_DECLINED; - } - - break; -#endif - - default: /* AF_INET */ - - sin1 = (struct sockaddr_in *) sa1; - sin2 = (struct sockaddr_in *) sa2; - - if (sin1->sin_port != sin2->sin_port) { - return NGX_DECLINED; - } - - if (sin1->sin_addr.s_addr != sin2->sin_addr.s_addr) { - return NGX_DECLINED; - } - - break; - } - - return NGX_OK; -} - - -static ngx_int_t ngx_init_zone_pool(ngx_cycle_t *cycle, ngx_shm_zone_t *zn) { u_char *file; diff -r ab493c60d9ff -r d39ef821d03e src/core/ngx_inet.c --- a/src/core/ngx_inet.c Fri Dec 06 14:30:27 2013 +0400 +++ b/src/core/ngx_inet.c Fri Dec 06 14:30:27 2013 +0400 @@ -1196,3 +1196,76 @@ ngx_inet_resolve_host(ngx_pool_t *pool, } #endif /* NGX_HAVE_GETADDRINFO && NGX_HAVE_INET6 */ + + +ngx_int_t +ngx_cmp_sockaddr(struct sockaddr *sa1, socklen_t slen1, + struct sockaddr *sa2, socklen_t slen2, ngx_uint_t cmp_port) +{ + struct sockaddr_in *sin1, *sin2; +#if (NGX_HAVE_INET6) + struct sockaddr_in6 *sin61, *sin62; +#endif +#if (NGX_HAVE_UNIX_DOMAIN) + struct sockaddr_un *saun1, *saun2; +#endif + + if (sa1->sa_family != sa2->sa_family) { + return NGX_DECLINED; + } + + switch (sa1->sa_family) { + +#if (NGX_HAVE_INET6) + case AF_INET6: + + sin61 = (struct sockaddr_in6 *) sa1; + sin62 = (struct sockaddr_in6 *) sa2; + + if (cmp_port && sin61->sin6_port != sin62->sin6_port) { + return NGX_DECLINED; + } + + if (ngx_memcmp(&sin61->sin6_addr, &sin62->sin6_addr, 16) != 0) { + return NGX_DECLINED; + } + + break; +#endif + +#if (NGX_HAVE_UNIX_DOMAIN) + case AF_UNIX: + + /* TODO length */ + + saun1 = (struct sockaddr_un *) sa1; + saun2 = (struct sockaddr_un *) sa2; + + if (ngx_memcmp(&saun1->sun_path, &saun2->sun_path, + sizeof(saun1->sun_path)) + != 0) + { + return NGX_DECLINED; + } + + break; +#endif + + default: /* AF_INET */ + + sin1 = (struct sockaddr_in *) sa1; + sin2 = (struct sockaddr_in *) sa2; + + if (cmp_port && sin1->sin_port != sin2->sin_port) { + return NGX_DECLINED; + } + + if (sin1->sin_addr.s_addr != sin2->sin_addr.s_addr) { + return NGX_DECLINED; + } + + break; + } + + return NGX_OK; +} diff -r ab493c60d9ff -r d39ef821d03e src/core/ngx_inet.h --- a/src/core/ngx_inet.h Fri Dec 06 14:30:27 2013 +0400 +++ b/src/core/ngx_inet.h Fri Dec 06 14:30:27 2013 +0400 @@ -115,6 +115,8 @@ ngx_int_t ngx_parse_addr(ngx_pool_t *poo size_t len); ngx_int_t ngx_parse_url(ngx_pool_t *pool, ngx_url_t *u); ngx_int_t ngx_inet_resolve_host(ngx_pool_t *pool, ngx_url_t *u); +ngx_int_t ngx_cmp_sockaddr(struct sockaddr *sa1, socklen_t slen1, + struct sockaddr *sa2, socklen_t slen2, ngx_uint_t cmp_port); #endif /* _NGX_INET_H_INCLUDED_ */ From ru at nginx.com Fri Dec 13 16:54:44 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Fri, 13 Dec 2013 16:54:44 +0000 Subject: [nginx] Resolver: fixed indentation. Message-ID: details: http://hg.nginx.org/nginx/rev/b43b02bb54db branches: changeset: 5474:b43b02bb54db user: Ruslan Ermilov date: Fri Dec 13 19:22:44 2013 +0400 description: Resolver: fixed indentation. diffstat: src/core/ngx_resolver.c | 48 ++++++++++++++++++++++++------------------------ 1 files changed, 24 insertions(+), 24 deletions(-) diffs (79 lines): diff -r d39ef821d03e -r b43b02bb54db src/core/ngx_resolver.c --- a/src/core/ngx_resolver.c Fri Dec 06 14:30:27 2013 +0400 +++ b/src/core/ngx_resolver.c Fri Dec 13 19:22:44 2013 +0400 @@ -1235,11 +1235,11 @@ ngx_resolver_process_a(ngx_resolver_t *r /* unlock name mutex */ while (next) { - ctx = next; - ctx->state = code; - next = ctx->next; - - ctx->handler(ctx); + ctx = next; + ctx->state = code; + next = ctx->next; + + ctx->handler(ctx); } return; @@ -1425,14 +1425,14 @@ ngx_resolver_process_a(ngx_resolver_t *r /* unlock name mutex */ while (next) { - ctx = next; - ctx->state = NGX_OK; - ctx->naddrs = naddrs; - ctx->addrs = (naddrs == 1) ? &ctx->addr : addrs; - ctx->addr = addr; - next = ctx->next; - - ctx->handler(ctx); + ctx = next; + ctx->state = NGX_OK; + ctx->naddrs = naddrs; + ctx->addrs = (naddrs == 1) ? &ctx->addr : addrs; + ctx->addr = addr; + next = ctx->next; + + ctx->handler(ctx); } if (naddrs > 1) { @@ -1587,11 +1587,11 @@ ngx_resolver_process_ptr(ngx_resolver_t /* unlock addr mutex */ while (next) { - ctx = next; - ctx->state = code; - next = ctx->next; - - ctx->handler(ctx); + ctx = next; + ctx->state = code; + next = ctx->next; + + ctx->handler(ctx); } return; @@ -1673,12 +1673,12 @@ ngx_resolver_process_ptr(ngx_resolver_t /* unlock addr mutex */ while (next) { - ctx = next; - ctx->state = NGX_OK; - ctx->name = name; - next = ctx->next; - - ctx->handler(ctx); + ctx = next; + ctx->state = NGX_OK; + ctx->name = name; + next = ctx->next; + + ctx->handler(ctx); } ngx_resolver_free(r, name.data); From ru at nginx.com Fri Dec 13 16:54:46 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Fri, 13 Dec 2013 16:54:46 +0000 Subject: [nginx] Changed resolver API to use ngx_addr_t. Message-ID: details: http://hg.nginx.org/nginx/rev/07dd5bd222ac branches: changeset: 5475:07dd5bd222ac user: Ruslan Ermilov date: Fri Dec 06 14:30:27 2013 +0400 description: Changed resolver API to use ngx_addr_t. diffstat: src/core/ngx_resolver.c | 240 +++++++++++++++++++----------- src/core/ngx_resolver.h | 5 +- src/event/ngx_event_openssl_stapling.c | 56 ++++-- src/http/ngx_http_upstream.c | 16 +- src/http/ngx_http_upstream.h | 2 +- src/http/ngx_http_upstream_round_robin.c | 40 +++-- src/mail/ngx_mail_smtp_handler.c | 43 ++-- 7 files changed, 245 insertions(+), 157 deletions(-) diffs (truncated from 738 to 300 lines): diff -r b43b02bb54db -r 07dd5bd222ac src/core/ngx_resolver.c --- a/src/core/ngx_resolver.c Fri Dec 13 19:22:44 2013 +0400 +++ b/src/core/ngx_resolver.c Fri Dec 06 14:30:27 2013 +0400 @@ -88,8 +88,8 @@ static void *ngx_resolver_calloc(ngx_res static void ngx_resolver_free(ngx_resolver_t *r, void *p); static void ngx_resolver_free_locked(ngx_resolver_t *r, void *p); static void *ngx_resolver_dup(ngx_resolver_t *r, void *src, size_t size); -static in_addr_t *ngx_resolver_rotate(ngx_resolver_t *r, in_addr_t *src, - ngx_uint_t n); +static ngx_addr_t *ngx_resolver_export(ngx_resolver_t *r, in_addr_t *src, + ngx_uint_t n, ngx_uint_t rotate); static u_char *ngx_resolver_log_error(ngx_log_t *log, u_char *buf, size_t len); @@ -281,7 +281,11 @@ ngx_resolve_start(ngx_resolver_t *r, ngx temp->state = NGX_OK; temp->naddrs = 1; temp->addrs = &temp->addr; - temp->addr = addr; + temp->addr.sockaddr = (struct sockaddr *) &temp->sin; + temp->addr.socklen = sizeof(struct sockaddr_in); + ngx_memzero(&temp->sin, sizeof(struct sockaddr_in)); + temp->sin.sin_family = AF_INET; + temp->sin.sin_addr.s_addr = addr; temp->quick = 1; return temp; @@ -417,9 +421,9 @@ static ngx_int_t ngx_resolve_name_locked(ngx_resolver_t *r, ngx_resolver_ctx_t *ctx) { uint32_t hash; - in_addr_t addr, *addrs; ngx_int_t rc; ngx_uint_t naddrs; + ngx_addr_t *addrs; ngx_resolver_ctx_t *next; ngx_resolver_node_t *rn; @@ -445,16 +449,14 @@ ngx_resolve_name_locked(ngx_resolver_t * /* NGX_RESOLVE_A answer */ - if (naddrs != 1) { - addr = 0; - addrs = ngx_resolver_rotate(r, rn->u.addrs, naddrs); + if (naddrs == 1) { + addrs = NULL; + + } else { + addrs = ngx_resolver_export(r, rn->u.addrs, naddrs, 1); if (addrs == NULL) { return NGX_ERROR; } - - } else { - addr = rn->u.addr; - addrs = NULL; } ctx->next = rn->waiting; @@ -465,8 +467,19 @@ ngx_resolve_name_locked(ngx_resolver_t * do { ctx->state = NGX_OK; ctx->naddrs = naddrs; - ctx->addrs = (naddrs == 1) ? &ctx->addr : addrs; - ctx->addr = addr; + + if (addrs == NULL) { + ctx->addrs = &ctx->addr; + ctx->addr.sockaddr = (struct sockaddr *) &ctx->sin; + ctx->addr.socklen = sizeof(struct sockaddr_in); + ngx_memzero(&ctx->sin, sizeof(struct sockaddr_in)); + ctx->sin.sin_family = AF_INET; + ctx->sin.sin_addr.s_addr = rn->u.addr; + + } else { + ctx->addrs = addrs; + } + next = ctx->next; ctx->handler(ctx); @@ -474,7 +487,8 @@ ngx_resolve_name_locked(ngx_resolver_t * ctx = next; } while (ctx); - if (addrs) { + if (addrs != NULL) { + ngx_resolver_free(r, addrs->sockaddr); ngx_resolver_free(r, addrs); } @@ -626,20 +640,29 @@ failed: } +/* AF_INET only */ + ngx_int_t ngx_resolve_addr(ngx_resolver_ctx_t *ctx) { u_char *name; + in_addr_t addr; ngx_resolver_t *r; + struct sockaddr_in *sin; ngx_resolver_node_t *rn; r = ctx->resolver; - ctx->addr = ntohl(ctx->addr); + if (ctx->addr.sockaddr->sa_family != AF_INET) { + return NGX_ERROR; + } + + sin = (struct sockaddr_in *) ctx->addr.sockaddr; + addr = ntohl(sin->sin_addr.s_addr); /* lock addr mutex */ - rn = ngx_resolver_lookup_addr(r, ctx->addr); + rn = ngx_resolver_lookup_addr(r, addr); if (rn) { @@ -694,7 +717,7 @@ ngx_resolve_addr(ngx_resolver_ctx_t *ctx goto failed; } - rn->node.key = ctx->addr; + rn->node.key = addr; rn->query = NULL; ngx_rbtree_insert(&r->addr_rbtree, &rn->node); @@ -765,14 +788,24 @@ failed: } +/* AF_INET only */ + void ngx_resolve_addr_done(ngx_resolver_ctx_t *ctx) { in_addr_t addr; ngx_resolver_t *r; ngx_resolver_ctx_t *w, **p; + struct sockaddr_in *sin; ngx_resolver_node_t *rn; + if (ctx->addr.sockaddr->sa_family != AF_INET) { + return; + } + + sin = (struct sockaddr_in *) ctx->addr.sockaddr; + addr = ntohl(sin->sin_addr.s_addr); + r = ctx->resolver; ngx_log_debug1(NGX_LOG_DEBUG_CORE, r->log, 0, @@ -786,7 +819,7 @@ ngx_resolve_addr_done(ngx_resolver_ctx_t if (ctx->state == NGX_AGAIN || ctx->state == NGX_RESOLVE_TIMEDOUT) { - rn = ngx_resolver_lookup_addr(r, ctx->addr); + rn = ngx_resolver_lookup_addr(r, addr); if (rn) { p = &rn->waiting; @@ -804,8 +837,6 @@ ngx_resolve_addr_done(ngx_resolver_ctx_t } } - addr = ntohl(ctx->addr); - ngx_log_error(NGX_LOG_ALERT, r->log, 0, "could not cancel %ud.%ud.%ud.%ud resolving", (addr >> 24) & 0xff, (addr >> 16) & 0xff, @@ -1177,8 +1208,9 @@ ngx_resolver_process_a(ngx_resolver_t *r size_t len; int32_t ttl; uint32_t hash; - in_addr_t addr, *addrs; + in_addr_t *addr; ngx_str_t name; + ngx_addr_t *addrs; ngx_uint_t type, class, qident, naddrs, a, i, n, start; ngx_resolver_an_t *an; ngx_resolver_ctx_t *ctx, *next; @@ -1247,8 +1279,6 @@ ngx_resolver_process_a(ngx_resolver_t *r i = ans; naddrs = 0; - addr = 0; - addrs = NULL; cname = NULL; ttl = 0; @@ -1319,9 +1349,6 @@ ngx_resolver_process_a(ngx_resolver_t *r goto short_response; } - addr = htonl((buf[i] << 24) + (buf[i + 1] << 16) - + (buf[i + 2] << 8) + (buf[i + 3])); - naddrs++; break; @@ -1352,66 +1379,69 @@ ngx_resolver_process_a(ngx_resolver_t *r if (naddrs) { if (naddrs == 1) { - rn->u.addr = addr; + addr = &rn->u.addr; + rn->naddrs = 1; } else { - - addrs = ngx_resolver_alloc(r, naddrs * sizeof(in_addr_t)); - if (addrs == NULL) { + addr = ngx_resolver_alloc(r, naddrs * sizeof(in_addr_t)); + if (addr == NULL) { goto failed; } - n = 0; - i = ans; - - for (a = 0; a < nan; a++) { - - for ( ;; ) { - - if (buf[i] & 0xc0) { - i += 2; - break; - } - - if (buf[i] == 0) { - i++; - break; - } - - i += 1 + buf[i]; + rn->u.addrs = addr; + rn->naddrs = (u_short) naddrs; + } + + n = 0; + i = ans; + + for (a = 0; a < nan; a++) { + + for ( ;; ) { + + if (buf[i] & 0xc0) { + i += 2; + break; } - an = (ngx_resolver_an_t *) &buf[i]; - - type = (an->type_hi << 8) + an->type_lo; - len = (an->len_hi << 8) + an->len_lo; - - i += sizeof(ngx_resolver_an_t); - - if (type == NGX_RESOLVE_A) { - - addrs[n++] = htonl((buf[i] << 24) + (buf[i + 1] << 16) - + (buf[i + 2] << 8) + (buf[i + 3])); - - if (n == naddrs) { - break; - } + if (buf[i] == 0) { + i++; + break; } - i += len; + i += 1 + buf[i]; } - rn->u.addrs = addrs; - - addrs = ngx_resolver_dup(r, rn->u.addrs, - naddrs * sizeof(in_addr_t)); + an = (ngx_resolver_an_t *) &buf[i]; + + type = (an->type_hi << 8) + an->type_lo; + len = (an->len_hi << 8) + an->len_lo; + + i += sizeof(ngx_resolver_an_t); + + if (type == NGX_RESOLVE_A) { + + addr[n] = htonl((buf[i] << 24) + (buf[i + 1] << 16) + + (buf[i + 2] << 8) + (buf[i + 3])); + + if (++n == naddrs) { + break; + } From ru at nginx.com Fri Dec 13 16:54:48 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Fri, 13 Dec 2013 16:54:48 +0000 Subject: [nginx] Resolver: implemented IPv6 address to name resolving. Message-ID: details: http://hg.nginx.org/nginx/rev/950c9ed3e66f branches: changeset: 5476:950c9ed3e66f user: Ruslan Ermilov date: Fri Dec 06 14:30:28 2013 +0400 description: Resolver: implemented IPv6 address to name resolving. diffstat: src/core/ngx_resolver.c | 484 +++++++++++++++++++++++++++++++++++++++-------- src/core/ngx_resolver.h | 12 + 2 files changed, 407 insertions(+), 89 deletions(-) diffs (truncated from 757 to 300 lines): diff -r 07dd5bd222ac -r 950c9ed3e66f src/core/ngx_resolver.c --- a/src/core/ngx_resolver.c Fri Dec 06 14:30:27 2013 +0400 +++ b/src/core/ngx_resolver.c Fri Dec 06 14:30:28 2013 +0400 @@ -92,6 +92,13 @@ static ngx_addr_t *ngx_resolver_export(n ngx_uint_t n, ngx_uint_t rotate); static u_char *ngx_resolver_log_error(ngx_log_t *log, u_char *buf, size_t len); +#if (NGX_HAVE_INET6) +static void ngx_resolver_rbtree_insert_addr6_value(ngx_rbtree_node_t *temp, + ngx_rbtree_node_t *node, ngx_rbtree_node_t *sentinel); +static ngx_resolver_node_t *ngx_resolver_lookup_addr6(ngx_resolver_t *r, + struct in6_addr *addr, uint32_t hash); +#endif + ngx_resolver_t * ngx_resolver_create(ngx_conf_t *cf, ngx_str_t *names, ngx_uint_t n) @@ -134,6 +141,15 @@ ngx_resolver_create(ngx_conf_t *cf, ngx_ ngx_queue_init(&r->name_expire_queue); ngx_queue_init(&r->addr_expire_queue); +#if (NGX_HAVE_INET6) + ngx_rbtree_init(&r->addr6_rbtree, &r->addr6_sentinel, + ngx_resolver_rbtree_insert_addr6_value); + + ngx_queue_init(&r->addr6_resend_queue); + + ngx_queue_init(&r->addr6_expire_queue); +#endif + r->event->handler = ngx_resolver_resend_handler; r->event->data = r; r->event->log = &cf->cycle->new_log; @@ -220,6 +236,10 @@ ngx_resolver_cleanup(void *data) ngx_resolver_cleanup_tree(r, &r->addr_rbtree); +#if (NGX_HAVE_INET6) + ngx_resolver_cleanup_tree(r, &r->addr6_rbtree); +#endif + if (r->event) { ngx_free(r->event); } @@ -640,30 +660,62 @@ failed: } -/* AF_INET only */ - ngx_int_t ngx_resolve_addr(ngx_resolver_ctx_t *ctx) { u_char *name; in_addr_t addr; + ngx_queue_t *resend_queue, *expire_queue; + ngx_rbtree_t *tree; ngx_resolver_t *r; struct sockaddr_in *sin; ngx_resolver_node_t *rn; +#if (NGX_HAVE_INET6) + uint32_t hash; + struct sockaddr_in6 *sin6; +#endif + +#if (NGX_SUPPRESS_WARN) + addr = 0; +#if (NGX_HAVE_INET6) + hash = 0; + sin6 = NULL; +#endif +#endif r = ctx->resolver; - if (ctx->addr.sockaddr->sa_family != AF_INET) { - return NGX_ERROR; + switch (ctx->addr.sockaddr->sa_family) { + +#if (NGX_HAVE_INET6) + case AF_INET6: + sin6 = (struct sockaddr_in6 *) ctx->addr.sockaddr; + hash = ngx_crc32_short(sin6->sin6_addr.s6_addr, 16); + + /* lock addr mutex */ + + rn = ngx_resolver_lookup_addr6(r, &sin6->sin6_addr, hash); + + tree = &r->addr6_rbtree; + resend_queue = &r->addr6_resend_queue; + expire_queue = &r->addr6_expire_queue; + + break; +#endif + + default: /* AF_INET */ + sin = (struct sockaddr_in *) ctx->addr.sockaddr; + addr = ntohl(sin->sin_addr.s_addr); + + /* lock addr mutex */ + + rn = ngx_resolver_lookup_addr(r, addr); + + tree = &r->addr_rbtree; + resend_queue = &r->addr_resend_queue; + expire_queue = &r->addr_expire_queue; } - sin = (struct sockaddr_in *) ctx->addr.sockaddr; - addr = ntohl(sin->sin_addr.s_addr); - - /* lock addr mutex */ - - rn = ngx_resolver_lookup_addr(r, addr); - if (rn) { if (rn->valid >= ngx_time()) { @@ -674,7 +726,7 @@ ngx_resolve_addr(ngx_resolver_ctx_t *ctx rn->expire = ngx_time() + r->expire; - ngx_queue_insert_head(&r->addr_expire_queue, &rn->queue); + ngx_queue_insert_head(expire_queue, &rn->queue); name = ngx_resolver_dup(r, rn->name, rn->nlen); if (name == NULL) { @@ -717,10 +769,22 @@ ngx_resolve_addr(ngx_resolver_ctx_t *ctx goto failed; } - rn->node.key = addr; + switch (ctx->addr.sockaddr->sa_family) { + +#if (NGX_HAVE_INET6) + case AF_INET6: + rn->addr6 = sin6->sin6_addr; + rn->node.key = hash; + break; +#endif + + default: /* AF_INET */ + rn->node.key = addr; + } + rn->query = NULL; - ngx_rbtree_insert(&r->addr_rbtree, &rn->node); + ngx_rbtree_insert(tree, &rn->node); } if (ngx_resolver_create_addr_query(rn, ctx) != NGX_OK) { @@ -743,13 +807,13 @@ ngx_resolve_addr(ngx_resolver_ctx_t *ctx ngx_add_timer(ctx->event, ctx->timeout); - if (ngx_queue_empty(&r->addr_resend_queue)) { + if (ngx_queue_empty(resend_queue)) { ngx_add_timer(r->event, (ngx_msec_t) (r->resend_timeout * 1000)); } rn->expire = ngx_time() + r->resend_timeout; - ngx_queue_insert_head(&r->addr_resend_queue, &rn->queue); + ngx_queue_insert_head(resend_queue, &rn->queue); rn->cnlen = 0; rn->naddrs = 0; @@ -767,7 +831,7 @@ ngx_resolve_addr(ngx_resolver_ctx_t *ctx failed: if (rn) { - ngx_rbtree_delete(&r->addr_rbtree, &rn->node); + ngx_rbtree_delete(tree, &rn->node); if (rn->query) { ngx_resolver_free(r, rn->query); @@ -788,26 +852,37 @@ failed: } -/* AF_INET only */ - void ngx_resolve_addr_done(ngx_resolver_ctx_t *ctx) { in_addr_t addr; + ngx_queue_t *expire_queue; + ngx_rbtree_t *tree; ngx_resolver_t *r; ngx_resolver_ctx_t *w, **p; struct sockaddr_in *sin; ngx_resolver_node_t *rn; - - if (ctx->addr.sockaddr->sa_family != AF_INET) { - return; +#if (NGX_HAVE_INET6) + uint32_t hash; + struct sockaddr_in6 *sin6; +#endif + + r = ctx->resolver; + + switch (ctx->addr.sockaddr->sa_family) { + +#if (NGX_HAVE_INET6) + case AF_INET6: + tree = &r->addr6_rbtree; + expire_queue = &r->addr6_expire_queue; + break; +#endif + + default: /* AF_INET */ + tree = &r->addr_rbtree; + expire_queue = &r->addr_expire_queue; } - sin = (struct sockaddr_in *) ctx->addr.sockaddr; - addr = ntohl(sin->sin_addr.s_addr); - - r = ctx->resolver; - ngx_log_debug1(NGX_LOG_DEBUG_CORE, r->log, 0, "resolve addr done: %i", ctx->state); @@ -819,7 +894,21 @@ ngx_resolve_addr_done(ngx_resolver_ctx_t if (ctx->state == NGX_AGAIN || ctx->state == NGX_RESOLVE_TIMEDOUT) { - rn = ngx_resolver_lookup_addr(r, addr); + switch (ctx->addr.sockaddr->sa_family) { + +#if (NGX_HAVE_INET6) + case AF_INET6: + sin6 = (struct sockaddr_in6 *) ctx->addr.sockaddr; + hash = ngx_crc32_short(sin6->sin6_addr.s6_addr, 16); + rn = ngx_resolver_lookup_addr6(r, &sin6->sin6_addr, hash); + break; +#endif + + default: /* AF_INET */ + sin = (struct sockaddr_in *) ctx->addr.sockaddr; + addr = ntohl(sin->sin_addr.s_addr); + rn = ngx_resolver_lookup_addr(r, addr); + } if (rn) { p = &rn->waiting; @@ -837,15 +926,22 @@ ngx_resolve_addr_done(ngx_resolver_ctx_t } } - ngx_log_error(NGX_LOG_ALERT, r->log, 0, - "could not cancel %ud.%ud.%ud.%ud resolving", - (addr >> 24) & 0xff, (addr >> 16) & 0xff, - (addr >> 8) & 0xff, addr & 0xff); + { + u_char text[NGX_SOCKADDR_STRLEN]; + ngx_str_t addrtext; + + addrtext.data = text; + addrtext.len = ngx_sock_ntop(ctx->addr.sockaddr, ctx->addr.socklen, + text, NGX_SOCKADDR_STRLEN, 0); + + ngx_log_error(NGX_LOG_ALERT, r->log, 0, + "could not cancel %V resolving", &addrtext); + } } done: - ngx_resolver_expire(r, &r->addr_rbtree, &r->addr_expire_queue); + ngx_resolver_expire(r, tree, expire_queue); /* unlock addr mutex */ @@ -946,6 +1042,9 @@ static void ngx_resolver_resend_handler(ngx_event_t *ev) { time_t timer, atimer, ntimer; +#if (NGX_HAVE_INET6) + time_t a6timer; +#endif ngx_resolver_t *r; r = ev->data; @@ -965,16 +1064,36 @@ ngx_resolver_resend_handler(ngx_event_t /* unlock addr mutex */ - if (ntimer == 0) { +#if (NGX_HAVE_INET6) + + /* lock addr6 mutex */ + + a6timer = ngx_resolver_resend(r, &r->addr6_rbtree, &r->addr6_resend_queue); + + /* unlock addr6 mutex */ + +#endif From ru at nginx.com Fri Dec 13 16:54:49 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Fri, 13 Dec 2013 16:54:49 +0000 Subject: [nginx] Resolver: implemented IPv6 name to address resolving. Message-ID: details: http://hg.nginx.org/nginx/rev/98876ce2a7fd branches: changeset: 5477:98876ce2a7fd user: Ruslan Ermilov date: Mon Dec 09 10:53:28 2013 +0400 description: Resolver: implemented IPv6 name to address resolving. diffstat: src/core/ngx_resolver.c | 487 ++++++++++++++++++++++++++++---- src/core/ngx_resolver.h | 18 +- src/event/ngx_event_openssl_stapling.c | 1 - src/http/ngx_http_upstream.c | 1 - src/mail/ngx_mail_smtp_handler.c | 1 - 5 files changed, 439 insertions(+), 69 deletions(-) diffs (truncated from 897 to 300 lines): diff -r 950c9ed3e66f -r 98876ce2a7fd src/core/ngx_resolver.c --- a/src/core/ngx_resolver.c Fri Dec 06 14:30:28 2013 +0400 +++ b/src/core/ngx_resolver.c Mon Dec 09 10:53:28 2013 +0400 @@ -70,7 +70,8 @@ static void ngx_resolver_read_response(n static void ngx_resolver_process_response(ngx_resolver_t *r, u_char *buf, size_t n); static void ngx_resolver_process_a(ngx_resolver_t *r, u_char *buf, size_t n, - ngx_uint_t ident, ngx_uint_t code, ngx_uint_t nan, ngx_uint_t ans); + ngx_uint_t ident, ngx_uint_t code, ngx_uint_t qtype, + ngx_uint_t nan, ngx_uint_t ans); static void ngx_resolver_process_ptr(ngx_resolver_t *r, u_char *buf, size_t n, ngx_uint_t ident, ngx_uint_t code, ngx_uint_t nan); static ngx_resolver_node_t *ngx_resolver_lookup_name(ngx_resolver_t *r, @@ -88,8 +89,8 @@ static void *ngx_resolver_calloc(ngx_res static void ngx_resolver_free(ngx_resolver_t *r, void *p); static void ngx_resolver_free_locked(ngx_resolver_t *r, void *p); static void *ngx_resolver_dup(ngx_resolver_t *r, void *src, size_t size); -static ngx_addr_t *ngx_resolver_export(ngx_resolver_t *r, in_addr_t *src, - ngx_uint_t n, ngx_uint_t rotate); +static ngx_addr_t *ngx_resolver_export(ngx_resolver_t *r, + ngx_resolver_node_t *rn, ngx_uint_t rotate); static u_char *ngx_resolver_log_error(ngx_log_t *log, u_char *buf, size_t len); #if (NGX_HAVE_INET6) @@ -435,8 +436,6 @@ done: } -/* NGX_RESOLVE_A only */ - static ngx_int_t ngx_resolve_name_locked(ngx_resolver_t *r, ngx_resolver_ctx_t *ctx) { @@ -463,17 +462,18 @@ ngx_resolve_name_locked(ngx_resolver_t * ngx_queue_insert_head(&r->name_expire_queue, &rn->queue); - naddrs = rn->naddrs; + naddrs = (rn->naddrs == (u_short) -1) ? 0 : rn->naddrs; +#if (NGX_HAVE_INET6) + naddrs += (rn->naddrs6 == (u_short) -1) ? 0 : rn->naddrs6; +#endif if (naddrs) { - /* NGX_RESOLVE_A answer */ - - if (naddrs == 1) { + if (naddrs == 1 && rn->naddrs == 1) { addrs = NULL; } else { - addrs = ngx_resolver_export(r, rn->u.addrs, naddrs, 1); + addrs = ngx_resolver_export(r, rn, 1); if (addrs == NULL) { return NGX_ERROR; } @@ -558,16 +558,25 @@ ngx_resolve_name_locked(ngx_resolver_t * if (rn->query) { ngx_resolver_free_locked(r, rn->query); rn->query = NULL; +#if (NGX_HAVE_INET6) + rn->query6 = NULL; +#endif } if (rn->cnlen) { ngx_resolver_free_locked(r, rn->u.cname); } - if (rn->naddrs > 1) { + if (rn->naddrs > 1 && rn->naddrs != (u_short) -1) { ngx_resolver_free_locked(r, rn->u.addrs); } +#if (NGX_HAVE_INET6) + if (rn->naddrs6 > 1 && rn->naddrs6 != (u_short) -1) { + ngx_resolver_free_locked(r, rn->u6.addrs6); + } +#endif + /* unlock alloc mutex */ } else { @@ -586,6 +595,9 @@ ngx_resolve_name_locked(ngx_resolver_t * rn->node.key = hash; rn->nlen = (u_short) ctx->name.len; rn->query = NULL; +#if (NGX_HAVE_INET6) + rn->query6 = NULL; +#endif ngx_rbtree_insert(&r->name_rbtree, &rn->node); } @@ -609,6 +621,11 @@ ngx_resolve_name_locked(ngx_resolver_t * return NGX_OK; } + rn->naddrs = (u_short) -1; +#if (NGX_HAVE_INET6) + rn->naddrs6 = (u_short) -1; +#endif + if (ngx_resolver_send_query(r, rn) != NGX_OK) { goto failed; } @@ -635,8 +652,8 @@ ngx_resolve_name_locked(ngx_resolver_t * ngx_queue_insert_head(&r->name_resend_queue, &rn->queue); + rn->code = 0; rn->cnlen = 0; - rn->naddrs = 0; rn->valid = 0; rn->waiting = ctx; @@ -762,6 +779,9 @@ ngx_resolve_addr(ngx_resolver_ctx_t *ctx ngx_resolver_free(r, rn->query); rn->query = NULL; +#if (NGX_HAVE_INET6) + rn->query6 = NULL; +#endif } else { rn = ngx_resolver_alloc(r, sizeof(ngx_resolver_node_t)); @@ -783,6 +803,9 @@ ngx_resolve_addr(ngx_resolver_ctx_t *ctx } rn->query = NULL; +#if (NGX_HAVE_INET6) + rn->query6 = NULL; +#endif ngx_rbtree_insert(tree, &rn->node); } @@ -791,6 +814,11 @@ ngx_resolve_addr(ngx_resolver_ctx_t *ctx goto failed; } + rn->naddrs = (u_short) -1; +#if (NGX_HAVE_INET6) + rn->naddrs6 = (u_short) -1; +#endif + if (ngx_resolver_send_query(r, rn) != NGX_OK) { goto failed; } @@ -815,8 +843,8 @@ ngx_resolve_addr(ngx_resolver_ctx_t *ctx ngx_queue_insert_head(resend_queue, &rn->queue); + rn->code = 0; rn->cnlen = 0; - rn->naddrs = 0; rn->name = NULL; rn->nlen = 0; rn->valid = 0; @@ -1023,16 +1051,33 @@ ngx_resolver_send_query(ngx_resolver_t * uc->connection->read->resolver = 1; } - n = ngx_send(uc->connection, rn->query, rn->qlen); - - if (n == -1) { - return NGX_ERROR; + if (rn->naddrs == (u_short) -1) { + n = ngx_send(uc->connection, rn->query, rn->qlen); + + if (n == -1) { + return NGX_ERROR; + } + + if ((size_t) n != (size_t) rn->qlen) { + ngx_log_error(NGX_LOG_CRIT, &uc->log, 0, "send() incomplete"); + return NGX_ERROR; + } } - if ((size_t) n != (size_t) rn->qlen) { - ngx_log_error(NGX_LOG_CRIT, &uc->log, 0, "send() incomplete"); - return NGX_ERROR; +#if (NGX_HAVE_INET6) + if (rn->query6 && rn->naddrs6 == (u_short) -1) { + n = ngx_send(uc->connection, rn->query6, rn->qlen); + + if (n == -1) { + return NGX_ERROR; + } + + if ((size_t) n != (size_t) rn->qlen) { + ngx_log_error(NGX_LOG_CRIT, &uc->log, 0, "send() incomplete"); + return NGX_ERROR; + } } +#endif return NGX_OK; } @@ -1174,6 +1219,9 @@ ngx_resolver_process_response(ngx_resolv char *err; ngx_uint_t i, times, ident, qident, flags, code, nqs, nan, qtype, qclass; +#if (NGX_HAVE_INET6) + ngx_uint_t qident6; +#endif ngx_queue_t *q; ngx_resolver_qs_t *qs; ngx_resolver_hdr_t *response; @@ -1217,12 +1265,18 @@ ngx_resolver_process_response(ngx_resolv qident = (rn->query[0] << 8) + rn->query[1]; if (qident == ident) { - ngx_log_error(r->log_level, r->log, 0, - "DNS error (%ui: %s), query id:%ui, name:\"%*s\"", - code, ngx_resolver_strerror(code), ident, - rn->nlen, rn->name); - return; + goto dns_error_name; } + +#if (NGX_HAVE_INET6) + if (rn->query6) { + qident6 = (rn->query6[0] << 8) + rn->query6[1]; + + if (qident6 == ident) { + goto dns_error_name; + } + } +#endif } goto dns_error; @@ -1279,8 +1333,11 @@ found: switch (qtype) { case NGX_RESOLVE_A: - - ngx_resolver_process_a(r, buf, n, ident, code, nan, +#if (NGX_HAVE_INET6) + case NGX_RESOLVE_AAAA: +#endif + + ngx_resolver_process_a(r, buf, n, ident, code, qtype, nan, i + sizeof(ngx_resolver_qs_t)); break; @@ -1309,6 +1366,14 @@ done: return; +dns_error_name: + + ngx_log_error(r->log_level, r->log, 0, + "DNS error (%ui: %s), query id:%ui, name:\"%*s\"", + code, ngx_resolver_strerror(code), ident, + rn->nlen, rn->name); + return; + dns_error: ngx_log_error(r->log_level, r->log, 0, @@ -1320,7 +1385,8 @@ dns_error: static void ngx_resolver_process_a(ngx_resolver_t *r, u_char *buf, size_t last, - ngx_uint_t ident, ngx_uint_t code, ngx_uint_t nan, ngx_uint_t ans) + ngx_uint_t ident, ngx_uint_t code, ngx_uint_t qtype, + ngx_uint_t nan, ngx_uint_t ans) { char *err; u_char *cname; @@ -1331,6 +1397,9 @@ ngx_resolver_process_a(ngx_resolver_t *r ngx_str_t name; ngx_addr_t *addrs; ngx_uint_t type, class, qident, naddrs, a, i, n, start; +#if (NGX_HAVE_INET6) + struct in6_addr *addr6; +#endif ngx_resolver_an_t *an; ngx_resolver_ctx_t *ctx, *next; ngx_resolver_node_t *rn; @@ -1350,14 +1419,43 @@ ngx_resolver_process_a(ngx_resolver_t *r rn = ngx_resolver_lookup_name(r, &name, hash); - if (rn == NULL || rn->query == NULL) { + if (rn == NULL) { ngx_log_error(r->log_level, r->log, 0, "unexpected response for %V", &name); ngx_resolver_free(r, name.data); goto failed; } - qident = (rn->query[0] << 8) + rn->query[1]; + switch (qtype) { + +#if (NGX_HAVE_INET6) + case NGX_RESOLVE_AAAA: + From ru at nginx.com Fri Dec 13 16:54:50 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Fri, 13 Dec 2013 16:54:50 +0000 Subject: [nginx] The "ipv6=" boolean parameter of the "resolver" directive. Message-ID: details: http://hg.nginx.org/nginx/rev/3cb3175a6fef branches: changeset: 5478:3cb3175a6fef user: Ruslan Ermilov date: Mon Dec 09 10:53:30 2013 +0400 description: The "ipv6=" boolean parameter of the "resolver" directive. When set to "off", only IPv4 addresses will be resolved, and no AAAA queries are ever sent. diffstat: src/core/ngx_resolver.c | 38 +++++++++++++++++++++++++++++++++++--- src/core/ngx_resolver.h | 1 + 2 files changed, 36 insertions(+), 3 deletions(-) diffs (101 lines): diff -r 98876ce2a7fd -r 3cb3175a6fef src/core/ngx_resolver.c --- a/src/core/ngx_resolver.c Mon Dec 09 10:53:28 2013 +0400 +++ b/src/core/ngx_resolver.c Mon Dec 09 10:53:30 2013 +0400 @@ -143,6 +143,8 @@ ngx_resolver_create(ngx_conf_t *cf, ngx_ ngx_queue_init(&r->addr_expire_queue); #if (NGX_HAVE_INET6) + r->ipv6 = 1; + ngx_rbtree_init(&r->addr6_rbtree, &r->addr6_sentinel, ngx_resolver_rbtree_insert_addr6_value); @@ -188,6 +190,25 @@ ngx_resolver_create(ngx_conf_t *cf, ngx_ continue; } +#if (NGX_HAVE_INET6) + if (ngx_strncmp(names[i].data, "ipv6=", 5) == 0) { + + if (ngx_strcmp(&names[i].data[5], "on") == 0) { + r->ipv6 = 1; + + } else if (ngx_strcmp(&names[i].data[5], "off") == 0) { + r->ipv6 = 0; + + } else { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "invalid parameter: %V", &names[i]); + return NULL; + } + + continue; + } +#endif + ngx_memzero(&u, sizeof(ngx_url_t)); u.url = names[i]; @@ -623,7 +644,7 @@ ngx_resolve_name_locked(ngx_resolver_t * rn->naddrs = (u_short) -1; #if (NGX_HAVE_INET6) - rn->naddrs6 = (u_short) -1; + rn->naddrs6 = r->ipv6 ? (u_short) -1 : 0; #endif if (ngx_resolver_send_query(r, rn) != NGX_OK) { @@ -2423,6 +2444,9 @@ ngx_resolver_create_name_query(ngx_resol u_char *p, *s; size_t len, nlen; ngx_uint_t ident; +#if (NGX_HAVE_INET6) + ngx_resolver_t *r; +#endif ngx_resolver_qs_t *qs; ngx_resolver_hdr_t *query; @@ -2431,7 +2455,9 @@ ngx_resolver_create_name_query(ngx_resol len = sizeof(ngx_resolver_hdr_t) + nlen + sizeof(ngx_resolver_qs_t); #if (NGX_HAVE_INET6) - p = ngx_resolver_alloc(ctx->resolver, len * 2); + r = ctx->resolver; + + p = ngx_resolver_alloc(ctx->resolver, r->ipv6 ? len * 2 : len); #else p = ngx_resolver_alloc(ctx->resolver, len); #endif @@ -2443,7 +2469,9 @@ ngx_resolver_create_name_query(ngx_resol rn->query = p; #if (NGX_HAVE_INET6) - rn->query6 = p + len; + if (r->ipv6) { + rn->query6 = p + len; + } #endif query = (ngx_resolver_hdr_t *) p; @@ -2509,6 +2537,10 @@ ngx_resolver_create_name_query(ngx_resol *p = (u_char) len; #if (NGX_HAVE_INET6) + if (!r->ipv6) { + return NGX_OK; + } + p = rn->query6; ngx_memcpy(p, rn->query, rn->qlen); diff -r 98876ce2a7fd -r 3cb3175a6fef src/core/ngx_resolver.h --- a/src/core/ngx_resolver.h Mon Dec 09 10:53:28 2013 +0400 +++ b/src/core/ngx_resolver.h Mon Dec 09 10:53:30 2013 +0400 @@ -123,6 +123,7 @@ typedef struct { ngx_queue_t addr_expire_queue; #if (NGX_HAVE_INET6) + ngx_uint_t ipv6; /* unsigned ipv6:1; */ ngx_rbtree_t addr6_rbtree; ngx_rbtree_node_t addr6_sentinel; ngx_queue_t addr6_resend_queue; From ru at nginx.com Fri Dec 13 16:54:52 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Fri, 13 Dec 2013 16:54:52 +0000 Subject: [nginx] Resolver: lookups are case-insensitive. Message-ID: details: http://hg.nginx.org/nginx/rev/c0d6eae5a1c5 branches: changeset: 5479:c0d6eae5a1c5 user: Ruslan Ermilov date: Fri Dec 13 20:49:52 2013 +0400 description: Resolver: lookups are case-insensitive. diffstat: src/core/ngx_resolver.c | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-) diffs (39 lines): diff -r 3cb3175a6fef -r c0d6eae5a1c5 src/core/ngx_resolver.c --- a/src/core/ngx_resolver.c Mon Dec 09 10:53:30 2013 +0400 +++ b/src/core/ngx_resolver.c Fri Dec 13 20:49:52 2013 +0400 @@ -467,6 +467,8 @@ ngx_resolve_name_locked(ngx_resolver_t * ngx_resolver_ctx_t *next; ngx_resolver_node_t *rn; + ngx_strlow(ctx->name.data, ctx->name.data, ctx->name.len); + hash = ngx_crc32_short(ctx->name.data, ctx->name.len); rn = ngx_resolver_lookup_name(r, &ctx->name, hash); @@ -2011,7 +2013,7 @@ ngx_resolver_process_ptr(ngx_resolver_t i += len; } - if (ngx_strcmp(&buf[i], "\7in-addr\4arpa") == 0) { + if (ngx_strcasecmp(&buf[i], (u_char *) "\7in-addr\4arpa") == 0) { i += sizeof("\7in-addr\4arpa"); /* lock addr mutex */ @@ -2058,7 +2060,7 @@ invalid_in_addr_arpa: addr6.s6_addr[octet] += (u_char) (digit * 16); } - if (ngx_strcmp(&buf[i], "\3ip6\4arpa") == 0) { + if (ngx_strcasecmp(&buf[i], (u_char *) "\3ip6\4arpa") == 0) { i += sizeof("\3ip6\4arpa"); /* lock addr mutex */ @@ -2737,7 +2739,7 @@ done: n = *src++; } else { - ngx_memcpy(dst, src, n); + ngx_strlow(dst, src, n); dst += n; src += n; From piotr at cloudflare.com Fri Dec 13 22:18:49 2013 From: piotr at cloudflare.com (Piotr Sikora) Date: Fri, 13 Dec 2013 14:18:49 -0800 Subject: [PATCH] SSL: support ALPN (IETF's successor to NPN) In-Reply-To: <5285F650.9050700@comodo.com> References: <201311140017.02611.vbart@nginx.com> <5285F650.9050700@comodo.com> Message-ID: Hey Valentin, OpenSSL-1.0.2 is around the corner [0], any chance of getting this in anytime soon? [0] http://git.openssl.org/gitweb/?p=openssl-web.git;a=commitdiff;h=866b44f57942b3d4250375af4861360eedcdd825 Best regards, Piotr Sikora From piotr at cloudflare.com Fri Dec 13 22:20:24 2013 From: piotr at cloudflare.com (Piotr Sikora) Date: Fri, 13 Dec 2013 14:20:24 -0800 Subject: [PATCH] SSL: support automatic selection of ECDH temporary key parameters In-Reply-To: References: Message-ID: Hey, OpenSSL-1.0.2 is around the corner [0], any chance of getting this in anytime soon? [0] http://git.openssl.org/gitweb/?p=openssl-web.git;a=commitdiff;h=866b44f57942b3d4250375af4861360eedcdd825 Best regards, Piotr Sikora From ru at nginx.com Mon Dec 16 09:52:25 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Mon, 16 Dec 2013 13:52:25 +0400 Subject: IPv6 support in resolver In-Reply-To: <20131112042341.GC5585@lo0.su> References: <20130617153021.GH72282@mdounin.ru> <51E3F300.3070509@nginx.com> <20131112042341.GC5585@lo0.su> Message-ID: <20131216095225.GP74021@lo0.su> On Tue, Nov 12, 2013 at 08:23:41AM +0400, Ruslan Ermilov wrote: > On Tue, Oct 29, 2013 at 04:06:35PM +0400, ToSHiC wrote: > > Yesterday you had a talk on Highload++ and said about lack of IPv6 resolver > > support. Do you have any news about my patch? > > This is just to let you know I've started to work on > adding IPv6 support into nginx's resolver. http://hg.nginx.org/nginx/rev/5c410d6ca7dd and the next 10 changesets added IPv6 support into nginx's resolver. From mdounin at mdounin.ru Tue Dec 17 13:59:49 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 17 Dec 2013 13:59:49 +0000 Subject: [nginx] nginx-1.5.8-RELEASE Message-ID: details: http://hg.nginx.org/nginx/rev/eaa76f249759 branches: changeset: 5480:eaa76f249759 user: Maxim Dounin date: Tue Dec 17 17:46:26 2013 +0400 description: nginx-1.5.8-RELEASE diffstat: docs/xml/nginx/changes.xml | 85 ++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 85 insertions(+), 0 deletions(-) diffs (95 lines): diff --git a/docs/xml/nginx/changes.xml b/docs/xml/nginx/changes.xml --- a/docs/xml/nginx/changes.xml +++ b/docs/xml/nginx/changes.xml @@ -5,6 +5,91 @@ + + + + +?????? resolver ???????????? IPv6. + + +IPv6 support in resolver. + + + + + +????????? listen ???????????? ???????? fastopen.
+??????? Mathew Rodley. +
+ +the "listen" directive supports the "fastopen" parameter.
+Thanks to Mathew Rodley. +
+
+ + + +????????? SSL ? ?????? ngx_http_uwsgi_module.
+??????? Roberto De Ioris. +
+ +SSL support in the ngx_http_uwsgi_module.
+Thanks to Roberto De Ioris. +
+
+ + + +??????? ????????? ?????????? ??? vim ????????? ? contrib.
+??????? Evan Miller. +
+ +vim syntax highlighting scripts were added to contrib.
+Thanks to Evan Miller. +
+
+ + + +??? ?????? ???? ??????? ? ?????????????? chunked transfer encoding +?? SSL-?????????? ??? ????????? ???????. + + +a timeout might occur while reading client request body +in an SSL connection using chunked transfer encoding. + + + + + +????????? master_process ???????? ??????????? ? nginx/Windows. + + +the "master_process" directive did not work correctly in nginx/Windows. + + + + + +???????? setfib ????????? listen ??? ?? ????????. + + +the "setfib" parameter of the "listen" directive might not work. + + + + + +? ?????? ngx_http_spdy_module. + + +in the ngx_http_spdy_module. + + + +
+ + From mdounin at mdounin.ru Tue Dec 17 13:59:50 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 17 Dec 2013 13:59:50 +0000 Subject: [nginx] release-1.5.8 tag Message-ID: details: http://hg.nginx.org/nginx/rev/c985f86d40b6 branches: changeset: 5481:c985f86d40b6 user: Maxim Dounin date: Tue Dec 17 17:46:26 2013 +0400 description: release-1.5.8 tag diffstat: .hgtags | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (8 lines): diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -363,3 +363,4 @@ 376a5e7694004048a9d073e4feb81bb54ee3ba91 60e0409b9ec7ee194c6d8102f0656598cc4a6cfe release-1.5.5 70c5cd3a61cb476c2afb3a61826e59c7cda0b7a7 release-1.5.6 9ba2542d75bf62a3972278c63561fc2ef5ec573a release-1.5.7 +eaa76f24975948b0ce8be01838d949122d44ed67 release-1.5.8 From wandenberg at gmail.com Tue Dec 17 23:05:16 2013 From: wandenberg at gmail.com (Wandenberg Peixoto) Date: Tue, 17 Dec 2013 21:05:16 -0200 Subject: Help with shared memory usage In-Reply-To: <20131006093708.GY62063@mdounin.ru> References: <20130701113629.GO20717@mdounin.ru> <20130729171109.GA2130@mdounin.ru> <20130730100931.GD2130@mdounin.ru> <20131006093708.GY62063@mdounin.ru> Message-ID: Hi Maxim, sorry for the long delay. I hope you remember my issue. In attach is the new patch with the changes you suggest. Can you check it again? I hope it can be applied to nginx code now. About this point "2. There is probably no need to check both prev and next.", I check both pointers to avoid a segmentation fault, since in some situations the next can be NULL and the prev may point to pool->free. As far as I could follow the code seems to me that could happen one of this pointers, next or prev, point to a NULL. I just made a double check to protect. If I'm wrong, just tell me what can I check to be sure the next and the previous free pages can be accessed, without problems and I will make the changes. Regards, Wandenberg On Sun, Oct 6, 2013 at 6:37 AM, Maxim Dounin wrote: > Hello! > > On Wed, Jul 31, 2013 at 12:28:02AM -0300, Wandenberg Peixoto wrote: > > > Hello! > > > > Thanks for your help. I hope that the patch be OK now. > > I don't know if the function and variable names are on nginx pattern. > > Feel free to change the patch. > > If you have any other point before accept it, will be a pleasure to fix > it. > > Sorry for long delay. As promised, I've looked into this. Apart > from multiple style problems, I see at least one case of possible use of > uninitialized memory. See below for comments. > > > > > --- src/core/ngx_slab.c 2013-05-06 07:27:10.000000000 -0300 > > +++ src/core/ngx_slab.c 2013-07-31 00:21:08.043034442 -0300 > > @@ -615,6 +615,26 @@ fail: > > > > > > static ngx_slab_page_t * > > +ngx_slab_merge_with_neighbour(ngx_slab_pool_t *pool, ngx_slab_page_t > *page) > > A side note: there are multiple issues with style in this patch. > In no particular order: > > 1. There is no reason to return ngx_slab_page_t * here. Just > ngx_int_t should be enough (or, even void would be good enough, > see below). > > 2. It is recommended do place function implementation below it's > use (and add a prototype). This allows to read the code linearly, > top-down. > > > +{ > > + ngx_slab_page_t *neighbour = &page[page->slab]; > > Here "neighbour" may point outside of the allocated page > structures if we are freeing the last page in the pool. > > > + if (((ngx_slab_page_t *) neighbour->prev != NULL) && > (neighbour->next > > != NULL) && ((neighbour->prev & NGX_SLAB_PAGE_MASK) == NGX_SLAB_PAGE)) { > > In no particular order: > > 1. Style problems - more than 80 chars, extra parens. > > 2. There is probably no need to check both prev and next. > > 3. See below about casting neighbour->prev to (ngx_slab_page_t *). > > > + page->slab += neighbour->slab; > > + > > + ((ngx_slab_page_t *) neighbour->prev)->next = neighbour->next; > > + neighbour->next->prev = neighbour->prev; > > Casting neighbour->prev to (ngx_slab_page_t *) without removing > last NGX_SLAB_PAGE_MASK bits isn't really a good idea. It works > here as NGX_SLAB_PAGE == 0, but may fail in other cases. > > It would be much better to use a snippet already used in other > places: > > prev = (ngx_slab_page_t *) (p->prev & ~NGX_SLAB_PAGE_MASK); > prev->next = p->next; > p->next->prev = p->prev; > > > + neighbour->slab = NGX_SLAB_PAGE_FREE; > > + neighbour->prev = (uintptr_t) &pool->free; > > + neighbour->next = &pool->free; > > Unused page structures used to be zeroed. Any reason for > prev/next pointing to pool->free? > > It looks like something like > > p->slab = NGX_SLAB_PAGE_FREE; > p->next = NULL; > p->prev = NGX_SLAB_PAGE; > > would be reasonable here. > > > + > > + return page; > > + } > > + return NULL; > > +} > > See above, there is no need to return page as it's unused. > Something like return NGX_OK / return NGX_DECLINED would be better > here - if we are going to check if any pages were actually merged. > > Additionally, an empty line should be before last return. > > > + > > + > > +static ngx_slab_page_t * > > ngx_slab_alloc_pages(ngx_slab_pool_t *pool, ngx_uint_t pages) > > { > > ngx_slab_page_t *page, *p; > > @@ -657,6 +677,19 @@ ngx_slab_alloc_pages(ngx_slab_pool_t *po > > } > > } > > > > + ngx_flag_t retry = 0; > > + for (page = pool->free.next; page != &pool->free;) { > > + if (ngx_slab_merge_with_neighbour(pool, page)) { > > + retry = 1; > > + } else { > > + page = page->next; > > + } > > + } > > + > > + if (retry) { > > + return ngx_slab_alloc_pages(pool, pages); > > + } > > + > > Apart from multiple style issues here, I think we should choose > one aproach: either loop over free pages and merge them here, or > maintain all free pages merged in ngx_slab_free_pages() (that is, > merge in both directions there). > > > ngx_slab_error(pool, NGX_LOG_CRIT, "ngx_slab_alloc() failed: no > > memory"); > > > > return NULL; > > @@ -687,6 +720,8 @@ ngx_slab_free_pages(ngx_slab_pool_t *poo > > page->next->prev = (uintptr_t) page; > > > > pool->free.next = page; > > + > > + ngx_slab_merge_with_neighbour(pool, page); > > } > > See above. > > Overral it looks much better than previous patch, but still needs > more work. > > -- > Maxim Dounin > http://nginx.org/en/donation.html > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: unfrag_slab_memory3.patch Type: text/x-patch Size: 1707 bytes Desc: not available URL: From rommer at activecloud.com Wed Dec 18 03:09:44 2013 From: rommer at activecloud.com (Raman Shishniou) Date: Wed, 18 Dec 2013 06:09:44 +0300 Subject: [PATCH v2] Make ngx_http_parse_unsafe_uri() to be able to unescape uri In-Reply-To: <52A7FCC6.1060503@activecloud.com> References: <52A3B1EA.3060203@activecloud.com> <20131210173203.GO95113@mdounin.ru> <52A7FCC6.1060503@activecloud.com> Message-ID: <52B111F8.6000100@activecloud.com> Hello, Is something wrong with this patch? On 12/11/2013 08:48 AM, Raman Shishniou wrote: > # HG changeset patch > # User Raman Shishniou > # Date 1386740673 -10800 > # Node ID 353840924c0ffd1203c8d0894f2128a2548f80c5 > # Parent a279d2a33dbfbad511f4415f833c35a60e46bb76 > Make ngx_http_parse_unsafe_uri() to be able to unescape uri > > It makes possible to use an escaped uri with "X-Accel-Redirect" header in > http_upstream module and "Destination" header http_dav module. No need > to unescape uri in http_ssi_filter module any more. > > diff --git a/src/http/modules/ngx_http_ssi_filter_module.c b/src/http/modules/ngx_http_ssi_filter_module.c > --- a/src/http/modules/ngx_http_ssi_filter_module.c > +++ b/src/http/modules/ngx_http_ssi_filter_module.c > @@ -1982,8 +1982,6 @@ > ngx_http_ssi_include(ngx_http_request_t *r, ngx_http_ssi_ctx_t *ctx, > ngx_str_t **params) > { > - u_char *dst, *src; > - size_t len; > ngx_int_t rc, key; > ngx_str_t *uri, *file, *wait, *set, *stub, args; > ngx_buf_t *b; > @@ -2054,18 +2052,6 @@ > return rc; > } > > - dst = uri->data; > - src = uri->data; > - > - ngx_unescape_uri(&dst, &src, uri->len, NGX_UNESCAPE_URI); > - > - len = (uri->data + uri->len) - src; > - if (len) { > - dst = ngx_movemem(dst, src, len); > - } > - > - uri->len = dst - uri->data; > - > ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, > "ssi include: \"%V\"", uri); > > diff --git a/src/http/ngx_http_parse.c b/src/http/ngx_http_parse.c > --- a/src/http/ngx_http_parse.c > +++ b/src/http/ngx_http_parse.c > @@ -9,6 +9,8 @@ > #include > #include > > +static ngx_inline ngx_int_t > +ngx_http_parse_test_doubledot(const u_char *p, const u_char *begin); > > static uint32_t usual[] = { > 0xffffdbfe, /* 1111 1111 1111 1111 1101 1011 1111 1110 */ > @@ -1776,35 +1778,75 @@ > } > > > +static ngx_inline ngx_int_t > +ngx_http_parse_test_doubledot(const u_char *p, const u_char *begin) > +{ > + /* assume *p is the path separator or > + p points to the next byte after the end */ > + > + if (p - 2 > begin && > + *(p - 1) == '.' && *(p - 2) == '.' && > + ngx_path_separator(*(p - 3))) { > + return 1; > + } > + > + if (p - 2 == begin && > + begin[0] == '.' && begin[1] == '.') { > + return 1; > + } > + > + return 0; > +} > + > + > ngx_int_t > ngx_http_parse_unsafe_uri(ngx_http_request_t *r, ngx_str_t *uri, > ngx_str_t *args, ngx_uint_t *flags) > { > - u_char ch, *p; > - size_t len; > - > + u_char *src, *dst, *newuri, ch, c, decoded; > + ngx_int_t unescape; > + size_t len; > + enum { > + sw_usual = 0, > + sw_quoted, > + sw_quoted_second > + } state; > + > + src = uri->data; > len = uri->len; > - p = uri->data; > - > - if (len == 0 || p[0] == '?') { > + unescape = 0; > + > + if (len == 0 || src[0] == '?') { > goto unsafe; > } > > - if (p[0] == '.' && len == 3 && p[1] == '.' && (ngx_path_separator(p[2]))) { > - goto unsafe; > - } > - > for ( /* void */ ; len; len--) { > > - ch = *p++; > + ch = *src++; > > if (usual[ch >> 5] & (1 << (ch & 0x1f))) { > continue; > } > > + if (ch == '%') { > + unescape++; > + continue; > + } > + > if (ch == '?') { > args->len = len - 1; > - args->data = p; > + args->data = src; > + > + if (unescape) { > + src--; > + goto unescape; > + } > + > + /* detect "/.." at the end or whole uri is ".." */ > + if (ngx_http_parse_test_doubledot(src - 1, uri->data)) { > + goto unsafe; > + } > + > uri->len -= len; > > return NGX_OK; > @@ -1814,16 +1856,151 @@ > goto unsafe; > } > > - if (ngx_path_separator(ch) && len > 2) { > - > - /* detect "/../" */ > - > - if (p[0] == '.' && p[1] == '.' && ngx_path_separator(p[2])) { > + /* detect "../" at the beginning or "/../" in the middle */ > + if (ngx_path_separator(ch) && > + ngx_http_parse_test_doubledot(src - 1, uri->data)) { > + goto unsafe; > + } > + } > + > + /* detect "/.." at the end or whole uri is ".." */ > + if (ngx_http_parse_test_doubledot(src, uri->data)) { > + goto unsafe; > + } > + > + if (!unescape) { > + return NGX_OK; > + } > + > +unescape: > + > + len = src - uri->data; > + newuri = ngx_pnalloc(r->pool, len); > + > + if (newuri == NULL) { > + return NGX_ERROR; > + } > + > + ngx_memcpy(newuri, uri->data, len); > + > + src = uri->data; > + dst = newuri; > + decoded = 0; > + state = 0; > + > + for ( /* void */ ; len; len--) { > + > + ch = *src++; > + > + switch (state) { > + > + case sw_usual: > + if (usual[ch >> 5] & (1 << (ch & 0x1f))) { > + *dst++ = ch; > + break; > + } > + > + if (ch == '%') { > + state = sw_quoted; > + break; > + } > + > + /* detect "../" at the beginning or "/../" in the middle */ > + if (ngx_path_separator(ch) && > + ngx_http_parse_test_doubledot(dst, newuri)) { > goto unsafe; > } > + > + *dst++ = ch; > + break; > + > + case sw_quoted: > + if (ch >= '0' && ch <= '9') { > + decoded = (u_char) (ch - '0'); > + state = sw_quoted_second; > + break; > + } > + > + c = (u_char) (ch | 0x20); > + if (c >= 'a' && c <= 'f') { > + decoded = (u_char) (c - 'a' + 10); > + state = sw_quoted_second; > + break; > + } > + > + if (ch == '%') { > + *dst++ = '%'; > + break; > + } > + > + *dst++ = '%'; > + *dst++ = ch; > + state = sw_usual; > + break; > + > + case sw_quoted_second: > + > + state = sw_usual; > + > + if (ch >= '0' && ch <= '9') { > + ch = (u_char) ((decoded << 4) + ch - '0'); > + > + if (ch == '\0') { > + goto unsafe; > + } > + > + *dst++ = ch; > + break; > + } > + > + c = (u_char) (ch | 0x20); > + if (c >= 'a' && c <= 'f') { > + ch = (u_char) ((decoded << 4) + c - 'a' + 10); > + > + /* detect "../" at the beginning or "/../" in the middle */ > + if (ngx_path_separator(ch) && > + ngx_http_parse_test_doubledot(dst, newuri)) { > + goto unsafe; > + } > + > + *dst++ = ch; > + break; > + } > + > + if (ch == '%') { > + *dst++ = '%'; > + *dst++ = *(src - 2); > + state = sw_quoted; > + break; > + } > + > + *dst++ = '%'; > + *dst++ = *(src - 2); > + *dst++ = ch; > + break; > } > } > > + switch (state) { > + case sw_usual: > + break; > + case sw_quoted: > + *dst++ = '%'; > + break; > + case sw_quoted_second: > + *dst++ = '%'; > + *dst++ = *(src - 1); > + break; > + } > + > + /* detect "/.." at the end or whole uri is ".." */ > + if (ngx_http_parse_test_doubledot(dst, newuri)) { > + goto unsafe; > + } > + > + uri->len = dst - newuri; > + uri->data = newuri; > + > return NGX_OK; > > unsafe: > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > From vbart at nginx.com Wed Dec 18 14:40:28 2013 From: vbart at nginx.com (Valentin Bartenev) Date: Wed, 18 Dec 2013 14:40:28 +0000 Subject: [nginx] Version bump. Message-ID: details: http://hg.nginx.org/nginx/rev/a1c65fa499e2 branches: changeset: 5482:a1c65fa499e2 user: Valentin Bartenev date: Wed Dec 18 18:37:48 2013 +0400 description: Version bump. diffstat: src/core/nginx.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (14 lines): diff -r c985f86d40b6 -r a1c65fa499e2 src/core/nginx.h --- a/src/core/nginx.h Tue Dec 17 17:46:26 2013 +0400 +++ b/src/core/nginx.h Wed Dec 18 18:37:48 2013 +0400 @@ -9,8 +9,8 @@ #define _NGINX_H_INCLUDED_ -#define nginx_version 1005008 -#define NGINX_VERSION "1.5.8" +#define nginx_version 1005009 +#define NGINX_VERSION "1.5.9" #define NGINX_VER "nginx/" NGINX_VERSION #define NGINX_VAR "NGINX" From vbart at nginx.com Wed Dec 18 14:40:30 2013 From: vbart at nginx.com (Valentin Bartenev) Date: Wed, 18 Dec 2013 14:40:30 +0000 Subject: [nginx] SPDY: use predefined constant for size of the Stream-ID ... Message-ID: details: http://hg.nginx.org/nginx/rev/86b563036d5a branches: changeset: 5483:86b563036d5a user: Valentin Bartenev date: Wed Dec 18 18:39:29 2013 +0400 description: SPDY: use predefined constant for size of the Stream-ID field. No functional changes. diffstat: src/http/ngx_http_spdy_filter_module.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r a1c65fa499e2 -r 86b563036d5a src/http/ngx_http_spdy_filter_module.c --- a/src/http/ngx_http_spdy_filter_module.c Wed Dec 18 18:37:48 2013 +0400 +++ b/src/http/ngx_http_spdy_filter_module.c Wed Dec 18 18:39:29 2013 +0400 @@ -751,7 +751,7 @@ ngx_http_spdy_filter_get_data_frame(ngx_ p = buf->start; buf->pos = p; - p += sizeof(uint32_t); + p += NGX_SPDY_SID_SIZE; (void) ngx_spdy_frame_write_flags_and_len(p, flags, len); From mdounin at mdounin.ru Wed Dec 18 15:26:26 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 18 Dec 2013 19:26:26 +0400 Subject: [PATCH v2] Make ngx_http_parse_unsafe_uri() to be able to unescape uri In-Reply-To: <52B111F8.6000100@activecloud.com> References: <52A3B1EA.3060203@activecloud.com> <20131210173203.GO95113@mdounin.ru> <52A7FCC6.1060503@activecloud.com> <52B111F8.6000100@activecloud.com> Message-ID: <20131218152626.GH95113@mdounin.ru> Hello! On Wed, Dec 18, 2013 at 06:09:44AM +0300, Raman Shishniou wrote: > Hello, > > Is something wrong with this patch? Previous statement that it looks overcomplicated still applies. [...] -- Maxim Dounin http://nginx.org/ From mdounin at mdounin.ru Thu Dec 19 12:07:51 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 19 Dec 2013 12:07:51 +0000 Subject: [nginx] Typo fixed. Message-ID: details: http://hg.nginx.org/nginx/rev/dc7f5c184473 branches: changeset: 5484:dc7f5c184473 user: Maxim Dounin date: Thu Dec 19 13:50:23 2013 +0400 description: Typo fixed. diffstat: docs/xml/nginx/changes.xml | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (17 lines): diff --git a/docs/xml/nginx/changes.xml b/docs/xml/nginx/changes.xml --- a/docs/xml/nginx/changes.xml +++ b/docs/xml/nginx/changes.xml @@ -20336,11 +20336,11 @@ the compressed response encrypted by SSL -????? TCP_NODELAY, TCP_NOPSUH ? TCP_CORK, ??????????? ??? TCP ???????, +????? TCP_NODELAY, TCP_NOPUSH ? TCP_CORK, ??????????? ??? TCP ???????, ?? ???????????? ??? unix domain ???????. -the TCP-specific TCP_NODELAY, TCP_NOPSUH, and TCP_CORK options, +the TCP-specific TCP_NODELAY, TCP_NOPUSH, and TCP_CORK options, are not used for the unix domain sockets. From ru at nginx.com Thu Dec 19 17:19:54 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Thu, 19 Dec 2013 17:19:54 +0000 Subject: [nginx] Resolver: use minimum TTL for caching (ticket #329). Message-ID: details: http://hg.nginx.org/nginx/rev/8958656a8060 branches: changeset: 5485:8958656a8060 user: Ruslan Ermilov date: Mon Dec 16 19:12:23 2013 +0400 description: Resolver: use minimum TTL for caching (ticket #329). diffstat: src/core/ngx_resolver.c | 13 +++++++------ 1 files changed, 7 insertions(+), 6 deletions(-) diffs (65 lines): diff -r dc7f5c184473 -r 8958656a8060 src/core/ngx_resolver.c --- a/src/core/ngx_resolver.c Thu Dec 19 13:50:23 2013 +0400 +++ b/src/core/ngx_resolver.c Mon Dec 16 19:12:23 2013 +0400 @@ -678,6 +678,7 @@ ngx_resolve_name_locked(ngx_resolver_t * rn->code = 0; rn->cnlen = 0; rn->valid = 0; + rn->ttl = NGX_MAX_UINT32_VALUE; rn->waiting = ctx; ctx->state = NGX_AGAIN; @@ -871,6 +872,7 @@ ngx_resolve_addr(ngx_resolver_ctx_t *ctx rn->name = NULL; rn->nlen = 0; rn->valid = 0; + rn->ttl = NGX_MAX_UINT32_VALUE; rn->waiting = ctx; /* unlock addr mutex */ @@ -1574,7 +1576,6 @@ ngx_resolver_process_a(ngx_resolver_t *r i = ans; naddrs = 0; cname = NULL; - ttl = 0; for (a = 0; a < nan; a++) { @@ -1628,6 +1629,8 @@ ngx_resolver_process_a(ngx_resolver_t *r ttl = 0; } + rn->ttl = ngx_min(rn->ttl, (uint32_t) ttl); + i += sizeof(ngx_resolver_an_t); switch (type) { @@ -1694,8 +1697,8 @@ ngx_resolver_process_a(ngx_resolver_t *r } ngx_log_debug3(NGX_LOG_DEBUG_CORE, r->log, 0, - "resolver naddrs:%ui cname:%p ttl:%d", - naddrs, cname, ttl); + "resolver naddrs:%ui cname:%p ttl:%uD", + naddrs, cname, rn->ttl); if (naddrs) { @@ -1746,8 +1749,6 @@ ngx_resolver_process_a(ngx_resolver_t *r #endif } - rn->ttl = ttl; - n = 0; i = ans; @@ -1915,7 +1916,7 @@ ngx_resolver_process_a(ngx_resolver_t *r rn->cnlen = (u_short) name.len; rn->u.cname = name.data; - rn->valid = ngx_time() + (r->valid ? r->valid : ttl); + rn->valid = ngx_time() + (r->valid ? r->valid : (time_t) rn->ttl); rn->expire = ngx_time() + r->expire; ngx_queue_insert_head(&r->name_expire_queue, &rn->queue); From vl at nginx.com Fri Dec 20 06:21:02 2013 From: vl at nginx.com (Homutov Vladimir) Date: Fri, 20 Dec 2013 06:21:02 +0000 Subject: [nginx] Upstream: simplified peer selection loop in the "ip_hash... Message-ID: details: http://hg.nginx.org/nginx/rev/741aa3fde496 branches: changeset: 5486:741aa3fde496 user: Vladimir Homutov date: Mon Dec 09 13:43:27 2013 +0400 description: Upstream: simplified peer selection loop in the "ip_hash" module. Conditions for skipping ineligible peers are rewritten to make adding of new conditions simpler and be in line with the "round_robin" and "least_conn" modules. No functional changes. diffstat: src/http/modules/ngx_http_upstream_ip_hash_module.c | 46 ++++++++++++-------- 1 files changed, 28 insertions(+), 18 deletions(-) diffs (72 lines): diff -r 8958656a8060 -r 741aa3fde496 src/http/modules/ngx_http_upstream_ip_hash_module.c --- a/src/http/modules/ngx_http_upstream_ip_hash_module.c Mon Dec 16 19:12:23 2013 +0400 +++ b/src/http/modules/ngx_http_upstream_ip_hash_module.c Mon Dec 09 13:43:27 2013 +0400 @@ -197,33 +197,39 @@ ngx_http_upstream_get_ip_hash_peer(ngx_p n = p / (8 * sizeof(uintptr_t)); m = (uintptr_t) 1 << p % (8 * sizeof(uintptr_t)); - if (!(iphp->rrp.tried[n] & m)) { + if (iphp->rrp.tried[n] & m) { + goto next; + } - ngx_log_debug2(NGX_LOG_DEBUG_HTTP, pc->log, 0, - "get ip hash peer, hash: %ui %04XA", p, m); + ngx_log_debug2(NGX_LOG_DEBUG_HTTP, pc->log, 0, + "get ip hash peer, hash: %ui %04XA", p, m); - peer = &iphp->rrp.peers->peer[p]; + peer = &iphp->rrp.peers->peer[p]; - /* ngx_lock_mutex(iphp->rrp.peers->mutex); */ + /* ngx_lock_mutex(iphp->rrp.peers->mutex); */ - if (!peer->down) { + if (peer->down) { + goto next_try; + } - if (peer->max_fails == 0 || peer->fails < peer->max_fails) { - break; - } + if (peer->max_fails + && peer->fails >= peer->max_fails + && now - peer->checked <= peer->fail_timeout) + { + goto next_try; + } - if (now - peer->checked > peer->fail_timeout) { - peer->checked = now; - break; - } - } + break; - iphp->rrp.tried[n] |= m; + next_try: - /* ngx_unlock_mutex(iphp->rrp.peers->mutex); */ + iphp->rrp.tried[n] |= m; - pc->tries--; - } + /* ngx_unlock_mutex(iphp->rrp.peers->mutex); */ + + pc->tries--; + + next: if (++iphp->tries >= 20) { return iphp->get_rr_peer(pc, &iphp->rrp); @@ -236,6 +242,10 @@ ngx_http_upstream_get_ip_hash_peer(ngx_p pc->socklen = peer->socklen; pc->name = &peer->name; + if (now - peer->checked > peer->fail_timeout) { + peer->checked = now; + } + /* ngx_unlock_mutex(iphp->rrp.peers->mutex); */ iphp->rrp.tried[n] |= m; From mdounin at mdounin.ru Fri Dec 20 12:19:07 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 20 Dec 2013 12:19:07 +0000 Subject: [nginx] SSL: ssl_buffer_size directive. Message-ID: details: http://hg.nginx.org/nginx/rev/a297b7ad6f94 branches: changeset: 5487:a297b7ad6f94 user: Maxim Dounin date: Fri Dec 20 16:18:25 2013 +0400 description: SSL: ssl_buffer_size directive. diffstat: src/event/ngx_event_openssl.c | 9 ++++++--- src/event/ngx_event_openssl.h | 2 ++ src/http/modules/ngx_http_ssl_module.c | 13 +++++++++++++ src/http/modules/ngx_http_ssl_module.h | 2 ++ 4 files changed, 23 insertions(+), 3 deletions(-) diffs (121 lines): diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c --- a/src/event/ngx_event_openssl.c +++ b/src/event/ngx_event_openssl.c @@ -190,6 +190,8 @@ ngx_ssl_create(ngx_ssl_t *ssl, ngx_uint_ return NGX_ERROR; } + ssl->buffer_size = NGX_SSL_BUFSIZE; + /* client side options */ SSL_CTX_set_options(ssl->ctx, SSL_OP_MICROSOFT_SESS_ID_BUG); @@ -726,6 +728,7 @@ ngx_ssl_create_connection(ngx_ssl_t *ssl } sc->buffer = ((flags & NGX_SSL_BUFFER) != 0); + sc->buffer_size = ssl->buffer_size; sc->connection = SSL_new(ssl->ctx); @@ -1222,7 +1225,7 @@ ngx_ssl_send_chain(ngx_connection_t *c, buf = c->ssl->buf; if (buf == NULL) { - buf = ngx_create_temp_buf(c->pool, NGX_SSL_BUFSIZE); + buf = ngx_create_temp_buf(c->pool, c->ssl->buffer_size); if (buf == NULL) { return NGX_CHAIN_ERROR; } @@ -1231,14 +1234,14 @@ ngx_ssl_send_chain(ngx_connection_t *c, } if (buf->start == NULL) { - buf->start = ngx_palloc(c->pool, NGX_SSL_BUFSIZE); + buf->start = ngx_palloc(c->pool, c->ssl->buffer_size); if (buf->start == NULL) { return NGX_CHAIN_ERROR; } buf->pos = buf->start; buf->last = buf->start; - buf->end = buf->start + NGX_SSL_BUFSIZE; + buf->end = buf->start + c->ssl->buffer_size; } send = buf->last - buf->pos; diff --git a/src/event/ngx_event_openssl.h b/src/event/ngx_event_openssl.h --- a/src/event/ngx_event_openssl.h +++ b/src/event/ngx_event_openssl.h @@ -29,6 +29,7 @@ typedef struct { SSL_CTX *ctx; ngx_log_t *log; + size_t buffer_size; } ngx_ssl_t; @@ -37,6 +38,7 @@ typedef struct { ngx_int_t last; ngx_buf_t *buf; + size_t buffer_size; ngx_connection_handler_pt handler; diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c --- a/src/http/modules/ngx_http_ssl_module.c +++ b/src/http/modules/ngx_http_ssl_module.c @@ -111,6 +111,13 @@ static ngx_command_t ngx_http_ssl_comma offsetof(ngx_http_ssl_srv_conf_t, ciphers), NULL }, + { ngx_string("ssl_buffer_size"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_TAKE1, + ngx_conf_set_size_slot, + NGX_HTTP_SRV_CONF_OFFSET, + offsetof(ngx_http_ssl_srv_conf_t, buffer_size), + NULL }, + { ngx_string("ssl_verify_client"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_TAKE1, ngx_conf_set_enum_slot, @@ -424,6 +431,7 @@ ngx_http_ssl_create_srv_conf(ngx_conf_t sscf->enable = NGX_CONF_UNSET; sscf->prefer_server_ciphers = NGX_CONF_UNSET; + sscf->buffer_size = NGX_CONF_UNSET_SIZE; sscf->verify = NGX_CONF_UNSET_UINT; sscf->verify_depth = NGX_CONF_UNSET_UINT; sscf->builtin_session_cache = NGX_CONF_UNSET; @@ -465,6 +473,9 @@ ngx_http_ssl_merge_srv_conf(ngx_conf_t * (NGX_CONF_BITMASK_SET|NGX_SSL_SSLv3|NGX_SSL_TLSv1 |NGX_SSL_TLSv1_1|NGX_SSL_TLSv1_2)); + ngx_conf_merge_size_value(conf->buffer_size, prev->buffer_size, + NGX_SSL_BUFSIZE); + ngx_conf_merge_uint_value(conf->verify, prev->verify, 0); ngx_conf_merge_uint_value(conf->verify_depth, prev->verify_depth, 1); @@ -572,6 +583,8 @@ ngx_http_ssl_merge_srv_conf(ngx_conf_t * return NGX_CONF_ERROR; } + conf->ssl.buffer_size = conf->buffer_size; + if (conf->verify) { if (conf->client_certificate.len == 0 && conf->verify != 3) { diff --git a/src/http/modules/ngx_http_ssl_module.h b/src/http/modules/ngx_http_ssl_module.h --- a/src/http/modules/ngx_http_ssl_module.h +++ b/src/http/modules/ngx_http_ssl_module.h @@ -26,6 +26,8 @@ typedef struct { ngx_uint_t verify; ngx_uint_t verify_depth; + size_t buffer_size; + ssize_t builtin_session_cache; time_t session_timeout; From ru at nginx.com Fri Dec 20 16:26:59 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Fri, 20 Dec 2013 20:26:59 +0400 Subject: [PATCH v2] Make ngx_http_parse_unsafe_uri() to be able to unescape uri In-Reply-To: <20131218152626.GH95113@mdounin.ru> References: <52A3B1EA.3060203@activecloud.com> <20131210173203.GO95113@mdounin.ru> <52A7FCC6.1060503@activecloud.com> <52B111F8.6000100@activecloud.com> <20131218152626.GH95113@mdounin.ru> Message-ID: <20131220162659.GQ23151@lo0.su> On Wed, Dec 18, 2013 at 07:26:26PM +0400, Maxim Dounin wrote: > On Wed, Dec 18, 2013 at 06:09:44AM +0300, Raman Shishniou wrote: > > Is something wrong with this patch? > > Previous statement that it looks overcomplicated still applies. > > [...] We've started internal testing of the attached patch series. You can help by testing and optionally reviewing it. Note it doesn't attempt to address the "../blabla" issue, and other similar issues. It's unclear at this point what should be the right fix. It's clear to me that the "/blabla/.." case should be considered unsafe. But as Maxim noted privately, we should probably consider all URIs not starting from "/" to be unsafe. -------------- next part -------------- # HG changeset patch # User Ruslan Ermilov # Date 1387556000 -14400 # Fri Dec 20 20:13:20 2013 +0400 # Node ID ef3ae992a4e34cbf725a570529d7e5c69d42afe1 # Parent d448241a2658ba3613719844f25f9666c50ffca5 Upstream: keep $upstream_http_x_accel_redirect intact. When processing the X-Accel-Redirect header, the value of the $upstream_http_x_accel_redirect variable was also overwritten. diff --git a/src/http/ngx_http_upstream.c b/src/http/ngx_http_upstream.c --- a/src/http/ngx_http_upstream.c +++ b/src/http/ngx_http_upstream.c @@ -1994,7 +1994,7 @@ ngx_http_upstream_test_connect(ngx_conne static ngx_int_t ngx_http_upstream_process_headers(ngx_http_request_t *r, ngx_http_upstream_t *u) { - ngx_str_t *uri, args; + ngx_str_t uri, args; ngx_uint_t i, flags; ngx_list_part_t *part; ngx_table_elt_t *h; @@ -2035,11 +2035,11 @@ ngx_http_upstream_process_headers(ngx_ht } } - uri = &u->headers_in.x_accel_redirect->value; + uri = u->headers_in.x_accel_redirect->value; ngx_str_null(&args); flags = NGX_HTTP_LOG_UNSAFE; - if (ngx_http_parse_unsafe_uri(r, uri, &args, &flags) != NGX_OK) { + if (ngx_http_parse_unsafe_uri(r, &uri, &args, &flags) != NGX_OK) { ngx_http_finalize_request(r, NGX_HTTP_NOT_FOUND); return NGX_DONE; } @@ -2048,7 +2048,7 @@ ngx_http_upstream_process_headers(ngx_ht r->method = NGX_HTTP_GET; } - ngx_http_internal_redirect(r, uri, &args); + ngx_http_internal_redirect(r, &uri, &args); ngx_http_finalize_request(r, NGX_DONE); return NGX_DONE; } # HG changeset patch # User Ruslan Ermilov # Date 1387556039 -14400 # Fri Dec 20 20:13:59 2013 +0400 # Node ID d177ae8bbe6fc1164544f5cdf9a02d68611144fa # Parent ef3ae992a4e34cbf725a570529d7e5c69d42afe1 Fixed an off-by-one bug in ngx_http_parse_unsafe_uri(). diff --git a/src/http/ngx_http_parse.c b/src/http/ngx_http_parse.c --- a/src/http/ngx_http_parse.c +++ b/src/http/ngx_http_parse.c @@ -1814,7 +1814,7 @@ ngx_http_parse_unsafe_uri(ngx_http_reque goto unsafe; } - if (ngx_path_separator(ch) && len > 2) { + if (ngx_path_separator(ch) && len >= 4) { /* detect "/../" */ # HG changeset patch # User Ruslan Ermilov # Date 1387556050 -14400 # Fri Dec 20 20:14:10 2013 +0400 # Node ID e4908c30a324934e7409c4a8e2285974937f2a4f # Parent d177ae8bbe6fc1164544f5cdf9a02d68611144fa Teach ngx_http_parse_unsafe_uri() how to unescape URIs. This fixes handling of escaped URIs in X-Accel-Redirect (ticket #316), SSI (ticket #240), and DAV. diff --git a/src/http/modules/ngx_http_ssi_filter_module.c b/src/http/modules/ngx_http_ssi_filter_module.c --- a/src/http/modules/ngx_http_ssi_filter_module.c +++ b/src/http/modules/ngx_http_ssi_filter_module.c @@ -1982,8 +1982,6 @@ static ngx_int_t ngx_http_ssi_include(ngx_http_request_t *r, ngx_http_ssi_ctx_t *ctx, ngx_str_t **params) { - u_char *dst, *src; - size_t len; ngx_int_t rc, key; ngx_str_t *uri, *file, *wait, *set, *stub, args; ngx_buf_t *b; @@ -2054,18 +2052,6 @@ ngx_http_ssi_include(ngx_http_request_t return rc; } - dst = uri->data; - src = uri->data; - - ngx_unescape_uri(&dst, &src, uri->len, NGX_UNESCAPE_URI); - - len = (uri->data + uri->len) - src; - if (len) { - dst = ngx_movemem(dst, src, len); - } - - uri->len = dst - uri->data; - ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "ssi include: \"%V\"", uri); diff --git a/src/http/ngx_http_parse.c b/src/http/ngx_http_parse.c --- a/src/http/ngx_http_parse.c +++ b/src/http/ngx_http_parse.c @@ -1780,11 +1780,13 @@ ngx_int_t ngx_http_parse_unsafe_uri(ngx_http_request_t *r, ngx_str_t *uri, ngx_str_t *args, ngx_uint_t *flags) { - u_char ch, *p; - size_t len; + u_char ch, *p, *src, *dst; + size_t len; + ngx_uint_t escaped; len = uri->len; p = uri->data; + escaped = 0; if (len == 0 || p[0] == '?') { goto unsafe; @@ -1798,6 +1800,11 @@ ngx_http_parse_unsafe_uri(ngx_http_reque ch = *p++; + if (ch == '%') { + escaped = 1; + continue; + } + if (usual[ch >> 5] & (1 << (ch & 0x1f))) { continue; } @@ -1807,7 +1814,7 @@ ngx_http_parse_unsafe_uri(ngx_http_reque args->data = p; uri->len -= len; - return NGX_OK; + break; } if (ch == '\0') { @@ -1824,6 +1831,53 @@ ngx_http_parse_unsafe_uri(ngx_http_reque } } + if (escaped) { + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "escaped URI: \"%V\"", uri); + + src = uri->data; + + dst = ngx_pnalloc(r->pool, uri->len); + if (dst == NULL) { + return NGX_ERROR; + } + + uri->data = dst; + + ngx_unescape_uri(&dst, &src, uri->len, 0); + + uri->len = dst - uri->data; + + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "unescaped URI: \"%V\"", uri); + + len = uri->len; + p = uri->data; + + if (p[0] == '.' && len == 3 && p[1] == '.' && ngx_path_separator(p[2])) + { + goto unsafe; + } + + for ( /* void */ ; len; len--) { + + ch = *p++; + + if (ch == '\0') { + goto unsafe; + } + + if (ngx_path_separator(ch) && len >= 4) { + + /* detect "/../" */ + + if (p[0] == '.' && p[1] == '.' && ngx_path_separator(p[2])) { + goto unsafe; + } + } + } + } + return NGX_OK; unsafe: # HG changeset patch # User Ruslan Ermilov # Date 1387556093 -14400 # Fri Dec 20 20:14:53 2013 +0400 # Node ID 0cf4215c164f429c9893014fc3dfbccde1120bbb # Parent e4908c30a324934e7409c4a8e2285974937f2a4f Dav: emit a warning about unsafe URI. diff --git a/src/http/modules/ngx_http_dav_module.c b/src/http/modules/ngx_http_dav_module.c --- a/src/http/modules/ngx_http_dav_module.c +++ b/src/http/modules/ngx_http_dav_module.c @@ -604,7 +604,7 @@ destination_done: duri.len = last - p; duri.data = p; - flags = 0; + flags = NGX_HTTP_LOG_UNSAFE; if (ngx_http_parse_unsafe_uri(r, &duri, &args, &flags) != NGX_OK) { goto invalid_destination; From mdounin at mdounin.ru Fri Dec 20 16:49:23 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 20 Dec 2013 20:49:23 +0400 Subject: Help with shared memory usage In-Reply-To: References: <20130701113629.GO20717@mdounin.ru> <20130729171109.GA2130@mdounin.ru> <20130730100931.GD2130@mdounin.ru> <20131006093708.GY62063@mdounin.ru> Message-ID: <20131220164923.GK95113@mdounin.ru> Hello! On Tue, Dec 17, 2013 at 09:05:16PM -0200, Wandenberg Peixoto wrote: > Hi Maxim, > > sorry for the long delay. I hope you remember my issue. > In attach is the new patch with the changes you suggest. > Can you check it again? I hope it can be applied to nginx code now. > > About this point "2. There is probably no need to check both prev and > next.", > I check both pointers to avoid a segmentation fault, since in some > situations the next can be NULL and the prev may point to pool->free. > As far as I could follow the code seems to me that could happen one of this > pointers, next or prev, point to a NULL. > I just made a double check to protect. As far as I see, all pages in the pool->free list are expected to have both p->prev and p->next set. And all pages with type NGX_SLAB_PAGE will be either on the pool->free list, or will have p->next set to NULL. [...] > > > +{ > > > + ngx_slab_page_t *neighbour = &page[page->slab]; > > > > Here "neighbour" may point outside of the allocated page > > structures if we are freeing the last page in the pool. It looks like you've tried to address this problem with the following check: > +static ngx_int_t > +ngx_slab_merge_pages(ngx_slab_pool_t *pool, ngx_slab_page_t *page) > +{ > + ngx_slab_page_t *prev, *p; > + > + p = &page[page->slab]; > + if ((u_char *) p >= pool->end) { > + return NGX_DECLINED; > + } This looks wrong, as pool->end points to the end of the pool, not the end of allocated page structures. -- Maxim Dounin http://nginx.org/ From igrigorik at gmail.com Fri Dec 20 18:58:43 2013 From: igrigorik at gmail.com (Ilya Grigorik) Date: Fri, 20 Dec 2013 10:58:43 -0800 Subject: [nginx] SSL: ssl_buffer_size directive. In-Reply-To: References: Message-ID: Awesome, really glad to see this! A couple of followup questions... (a) Is there any way to force a packet flush on record end? At the moment nginx will fragment multiple records across packet boundaries, which is suboptimal as it means that I need a minimum of two packets to decode any record - e.g. if I set my record size to 1370 bytes, the first packet will contain the first full record plus another 20-50 bytes of next record. (b) Current NGX_SSL_BUFSIZE is set to 16KB which is effectively guaranteed to overflow the CWND of a new connection and introduce another RTT for interactive traffic - e.g. HTTP pages. I would love to see a lower starting record size to mitigate this problem -- defaults matter! On the subject of optimizing record size, the GFE team at Google recently landed ~following logic: - new connections default to small record size -- each record fits into a TCP packet -- packets are flushed at record boundaries - server tracks number of bytes written since reset and timestamp of last write -- if bytes written > {configurable byte threshold) then boost record size to 16KB -- if last write timestamp > now - {configurable time threshold} then reset sent byte count In other words, start with small record size to optimize for delivery of small/interactive objects (bulk of HTTP traffic). Then, if large file is being transferred bump record size to 16KB and continue using that until the connection goes idle.. when communication resumes, start with small record size and repeat. Overall, this is aimed to optimize delivery of small files where incremental delivery is a priority, and also for large downloads where overall throughput is a priority. Both byte and time thresholds are exposed as configurable flags, and current defaults in GFE are 1MB and 1s. This would require a bit more work than the current patch, but I'd love to see a similar strategy in nginx. Hardcoding a fixed record size will inevitably lead to suboptimal delivery of either interactive or bulk traffic. Thoughts? ig On Fri, Dec 20, 2013 at 4:19 AM, Maxim Dounin wrote: > details: http://hg.nginx.org/nginx/rev/a297b7ad6f94 > branches: > changeset: 5487:a297b7ad6f94 > user: Maxim Dounin > date: Fri Dec 20 16:18:25 2013 +0400 > description: > SSL: ssl_buffer_size directive. > > diffstat: > > src/event/ngx_event_openssl.c | 9 ++++++--- > src/event/ngx_event_openssl.h | 2 ++ > src/http/modules/ngx_http_ssl_module.c | 13 +++++++++++++ > src/http/modules/ngx_http_ssl_module.h | 2 ++ > 4 files changed, 23 insertions(+), 3 deletions(-) > > diffs (121 lines): > > diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c > --- a/src/event/ngx_event_openssl.c > +++ b/src/event/ngx_event_openssl.c > @@ -190,6 +190,8 @@ ngx_ssl_create(ngx_ssl_t *ssl, ngx_uint_ > return NGX_ERROR; > } > > + ssl->buffer_size = NGX_SSL_BUFSIZE; > + > /* client side options */ > > SSL_CTX_set_options(ssl->ctx, SSL_OP_MICROSOFT_SESS_ID_BUG); > @@ -726,6 +728,7 @@ ngx_ssl_create_connection(ngx_ssl_t *ssl > } > > sc->buffer = ((flags & NGX_SSL_BUFFER) != 0); > + sc->buffer_size = ssl->buffer_size; > > sc->connection = SSL_new(ssl->ctx); > > @@ -1222,7 +1225,7 @@ ngx_ssl_send_chain(ngx_connection_t *c, > buf = c->ssl->buf; > > if (buf == NULL) { > - buf = ngx_create_temp_buf(c->pool, NGX_SSL_BUFSIZE); > + buf = ngx_create_temp_buf(c->pool, c->ssl->buffer_size); > if (buf == NULL) { > return NGX_CHAIN_ERROR; > } > @@ -1231,14 +1234,14 @@ ngx_ssl_send_chain(ngx_connection_t *c, > } > > if (buf->start == NULL) { > - buf->start = ngx_palloc(c->pool, NGX_SSL_BUFSIZE); > + buf->start = ngx_palloc(c->pool, c->ssl->buffer_size); > if (buf->start == NULL) { > return NGX_CHAIN_ERROR; > } > > buf->pos = buf->start; > buf->last = buf->start; > - buf->end = buf->start + NGX_SSL_BUFSIZE; > + buf->end = buf->start + c->ssl->buffer_size; > } > > send = buf->last - buf->pos; > diff --git a/src/event/ngx_event_openssl.h b/src/event/ngx_event_openssl.h > --- a/src/event/ngx_event_openssl.h > +++ b/src/event/ngx_event_openssl.h > @@ -29,6 +29,7 @@ > typedef struct { > SSL_CTX *ctx; > ngx_log_t *log; > + size_t buffer_size; > } ngx_ssl_t; > > > @@ -37,6 +38,7 @@ typedef struct { > > ngx_int_t last; > ngx_buf_t *buf; > + size_t buffer_size; > > ngx_connection_handler_pt handler; > > diff --git a/src/http/modules/ngx_http_ssl_module.c > b/src/http/modules/ngx_http_ssl_module.c > --- a/src/http/modules/ngx_http_ssl_module.c > +++ b/src/http/modules/ngx_http_ssl_module.c > @@ -111,6 +111,13 @@ static ngx_command_t ngx_http_ssl_comma > offsetof(ngx_http_ssl_srv_conf_t, ciphers), > NULL }, > > + { ngx_string("ssl_buffer_size"), > + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_TAKE1, > + ngx_conf_set_size_slot, > + NGX_HTTP_SRV_CONF_OFFSET, > + offsetof(ngx_http_ssl_srv_conf_t, buffer_size), > + NULL }, > + > { ngx_string("ssl_verify_client"), > NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_TAKE1, > ngx_conf_set_enum_slot, > @@ -424,6 +431,7 @@ ngx_http_ssl_create_srv_conf(ngx_conf_t > > sscf->enable = NGX_CONF_UNSET; > sscf->prefer_server_ciphers = NGX_CONF_UNSET; > + sscf->buffer_size = NGX_CONF_UNSET_SIZE; > sscf->verify = NGX_CONF_UNSET_UINT; > sscf->verify_depth = NGX_CONF_UNSET_UINT; > sscf->builtin_session_cache = NGX_CONF_UNSET; > @@ -465,6 +473,9 @@ ngx_http_ssl_merge_srv_conf(ngx_conf_t * > (NGX_CONF_BITMASK_SET|NGX_SSL_SSLv3|NGX_SSL_TLSv1 > |NGX_SSL_TLSv1_1|NGX_SSL_TLSv1_2)); > > + ngx_conf_merge_size_value(conf->buffer_size, prev->buffer_size, > + NGX_SSL_BUFSIZE); > + > ngx_conf_merge_uint_value(conf->verify, prev->verify, 0); > ngx_conf_merge_uint_value(conf->verify_depth, prev->verify_depth, 1); > > @@ -572,6 +583,8 @@ ngx_http_ssl_merge_srv_conf(ngx_conf_t * > return NGX_CONF_ERROR; > } > > + conf->ssl.buffer_size = conf->buffer_size; > + > if (conf->verify) { > > if (conf->client_certificate.len == 0 && conf->verify != 3) { > diff --git a/src/http/modules/ngx_http_ssl_module.h > b/src/http/modules/ngx_http_ssl_module.h > --- a/src/http/modules/ngx_http_ssl_module.h > +++ b/src/http/modules/ngx_http_ssl_module.h > @@ -26,6 +26,8 @@ typedef struct { > ngx_uint_t verify; > ngx_uint_t verify_depth; > > + size_t buffer_size; > + > ssize_t builtin_session_cache; > > time_t session_timeout; > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex at zeitgeist.se Fri Dec 20 20:49:02 2013 From: alex at zeitgeist.se (Alex) Date: Fri, 20 Dec 2013 21:49:02 +0100 Subject: [nginx] SSL: ssl_buffer_size directive. In-Reply-To: References: Message-ID: On 2013-12-20 13:19, Maxim Dounin wrote: > description: > SSL: ssl_buffer_size directive. Great to see this going into mainline. On 2013-12-20 19:58, Ilya Grigorik wrote: > (a) Is there any way to force a packet flush on record end? That would be indeed nice. Flushing would prevent a TLS record from spilling over into later TCP segments, which ensures that each encrypted packet payload can be decrypted completely per TCP segment. > This would require a bit more work than the current patch, but I'd love to see a similar strategy in nginx. Hardcoding a fixed record size will inevitably lead to suboptimal delivery of either interactive or bulk traffic. Thoughts? It'd be interesting to know how difficult it'd be to implement such a dynamic behavior of the SSL buffer size. An easier, albeit less optimal solution would be to adjust the ssl_buffer_size directive depending on the request URI (via location blocks). Not sure if Maxim's patch would allow for that already? If large files are served from a known request URI pattern, you could then increase the SSL buffer size accordingly for that location. From yaoweibin at gmail.com Sun Dec 22 12:56:35 2013 From: yaoweibin at gmail.com (Weibin Yao) Date: Sun, 22 Dec 2013 20:56:35 +0800 Subject: [PATCH] SPDY: support for SPDY v3 Message-ID: Hi, We have just implemented the support for SPDY v3 in nginx, with flow control (upload and download), and a switch option between SPDY v2 and SPDY v3. This patch is for Nginx-1.5.8. The directives we added are: spdy_version syntax: spdy_version [2|3] default: spdy_version 3 context: http, server Specify the version of current SPDY protocol. spdy_flow_control syntax: spdy_flow_control on|off default: spdy_flow_control on context: http, server Turn on or off with SPDY flow control. spdy_init_recv_window_size syntax: spdy_init_recv_window_size size default: spdy_init_recv_window_size 64k context: http, server Specify the receiving window size for SPDY. By default, it's 64K. It will send a WINDOW UPDATE frame when it receives half of the window size data every time. Hope this patch is helpful. Have fun :-) Thank you. -- Weibin Yao Developer @ Web Platform Team of Taobao -------------- next part -------------- A non-text attachment was scrubbed... Name: SPDY_v3.patch Type: application/octet-stream Size: 139773 bytes Desc: not available URL: From mdounin at mdounin.ru Sun Dec 22 22:27:36 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 23 Dec 2013 02:27:36 +0400 Subject: [nginx] SSL: ssl_buffer_size directive. In-Reply-To: References: Message-ID: <20131222222735.GT95113@mdounin.ru> Hello! On Fri, Dec 20, 2013 at 10:58:43AM -0800, Ilya Grigorik wrote: > Awesome, really glad to see this! A couple of followup questions... > > (a) Is there any way to force a packet flush on record end? At the moment > nginx will fragment multiple records across packet boundaries, which is > suboptimal as it means that I need a minimum of two packets to decode any > record - e.g. if I set my record size to 1370 bytes, the first packet will > contain the first full record plus another 20-50 bytes of next record. There is OpenSSL socket layer on the way down. It may be possible to achieve something with SSL_[CTX_]set_max_send_fragment() in OpenSSL 1.0.0+, but I haven't looked into details. (As I already said, I don't think that using packet-sized records is a good idea, it looks like an overkill and waste of resources, both network and CPU.) > (b) Current NGX_SSL_BUFSIZE is set to 16KB which is effectively guaranteed > to overflow the CWND of a new connection and introduce another RTT for > interactive traffic - e.g. HTTP pages. I would love to see a lower starting > record size to mitigate this problem -- defaults matter! We are considering using 4k or 8k as the default in the future. For now, the directive is mostly to simplify experimenting with various buffer sizes. > On the subject of optimizing record size, the GFE team at Google recently > landed ~following logic: > > - new connections default to small record size > -- each record fits into a TCP packet > -- packets are flushed at record boundaries > - server tracks number of bytes written since reset and timestamp of last > write > -- if bytes written > {configurable byte threshold) then boost record size > to 16KB > -- if last write timestamp > now - {configurable time threshold} then reset > sent byte count > > In other words, start with small record size to optimize for delivery of > small/interactive objects (bulk of HTTP traffic). Then, if large file is > being transferred bump record size to 16KB and continue using that until > the connection goes idle.. when communication resumes, start with small > record size and repeat. Overall, this is aimed to optimize delivery of > small files where incremental delivery is a priority, and also for large > downloads where overall throughput is a priority. > > Both byte and time thresholds are exposed as configurable flags, and > current defaults in GFE are 1MB and 1s. > > This would require a bit more work than the current patch, but I'd love to > see a similar strategy in nginx. Hardcoding a fixed record size will > inevitably lead to suboptimal delivery of either interactive or bulk > traffic. Thoughts? While some logic like this is certainly needed to use packet-sized records, it looks overcomplicated and probably not at all needed with 4k/8k buffers. -- Maxim Dounin http://nginx.org/ From ru at nginx.com Mon Dec 23 11:50:27 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Mon, 23 Dec 2013 11:50:27 +0000 Subject: [nginx] Fixed the first argument to getsockopt(). Message-ID: details: http://hg.nginx.org/nginx/rev/c51b9491b0bd branches: changeset: 5488:c51b9491b0bd user: Ruslan Ermilov date: Thu Dec 19 13:43:18 2013 +0400 description: Fixed the first argument to getsockopt(). While here, always initialize the last argument. diffstat: src/core/ngx_connection.c | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-) diffs (25 lines): diff -r a297b7ad6f94 -r c51b9491b0bd src/core/ngx_connection.c --- a/src/core/ngx_connection.c Fri Dec 20 16:18:25 2013 +0400 +++ b/src/core/ngx_connection.c Thu Dec 19 13:43:18 2013 +0400 @@ -199,7 +199,9 @@ ngx_set_inherited_sockets(ngx_cycle_t *c #if (NGX_HAVE_SETFIB) - if (getsockopt(ls[i].setfib, SOL_SOCKET, SO_SETFIB, + olen = sizeof(int); + + if (getsockopt(ls[i].fd, SOL_SOCKET, SO_SETFIB, (void *) &ls[i].setfib, &olen) == -1) { @@ -215,7 +217,9 @@ ngx_set_inherited_sockets(ngx_cycle_t *c #if (NGX_HAVE_TCP_FASTOPEN) - if (getsockopt(ls[i].fastopen, IPPROTO_TCP, TCP_FASTOPEN, + olen = sizeof(int); + + if (getsockopt(ls[i].fd, IPPROTO_TCP, TCP_FASTOPEN, (void *) &ls[i].fastopen, &olen) == -1) { From ru at nginx.com Mon Dec 23 14:12:42 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Mon, 23 Dec 2013 14:12:42 +0000 Subject: [nginx] Upstream: keep $upstream_http_x_accel_redirect intact. Message-ID: details: http://hg.nginx.org/nginx/rev/6d357b2a9d6e branches: changeset: 5489:6d357b2a9d6e user: Ruslan Ermilov date: Mon Dec 23 18:11:46 2013 +0400 description: Upstream: keep $upstream_http_x_accel_redirect intact. When processing the X-Accel-Redirect header, the value of the $upstream_http_x_accel_redirect variable was also overwritten. diffstat: src/http/ngx_http_upstream.c | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diffs (35 lines): diff -r c51b9491b0bd -r 6d357b2a9d6e src/http/ngx_http_upstream.c --- a/src/http/ngx_http_upstream.c Thu Dec 19 13:43:18 2013 +0400 +++ b/src/http/ngx_http_upstream.c Mon Dec 23 18:11:46 2013 +0400 @@ -1994,7 +1994,7 @@ ngx_http_upstream_test_connect(ngx_conne static ngx_int_t ngx_http_upstream_process_headers(ngx_http_request_t *r, ngx_http_upstream_t *u) { - ngx_str_t *uri, args; + ngx_str_t uri, args; ngx_uint_t i, flags; ngx_list_part_t *part; ngx_table_elt_t *h; @@ -2035,11 +2035,11 @@ ngx_http_upstream_process_headers(ngx_ht } } - uri = &u->headers_in.x_accel_redirect->value; + uri = u->headers_in.x_accel_redirect->value; ngx_str_null(&args); flags = NGX_HTTP_LOG_UNSAFE; - if (ngx_http_parse_unsafe_uri(r, uri, &args, &flags) != NGX_OK) { + if (ngx_http_parse_unsafe_uri(r, &uri, &args, &flags) != NGX_OK) { ngx_http_finalize_request(r, NGX_HTTP_NOT_FOUND); return NGX_DONE; } @@ -2048,7 +2048,7 @@ ngx_http_upstream_process_headers(ngx_ht r->method = NGX_HTTP_GET; } - ngx_http_internal_redirect(r, uri, &args); + ngx_http_internal_redirect(r, &uri, &args); ngx_http_finalize_request(r, NGX_DONE); return NGX_DONE; } From ru at nginx.com Mon Dec 23 14:12:43 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Mon, 23 Dec 2013 14:12:43 +0000 Subject: [nginx] Detect more unsafe URIs in ngx_http_parse_unsafe_uri(). Message-ID: details: http://hg.nginx.org/nginx/rev/b141a7627ac6 branches: changeset: 5490:b141a7627ac6 user: Ruslan Ermilov date: Mon Dec 23 18:11:56 2013 +0400 description: Detect more unsafe URIs in ngx_http_parse_unsafe_uri(). The following URIs were considered safe: "..", "../foo", and "/foo/..". diffstat: src/http/ngx_http_parse.c | 10 +++++++--- 1 files changed, 7 insertions(+), 3 deletions(-) diffs (28 lines): diff -r 6d357b2a9d6e -r b141a7627ac6 src/http/ngx_http_parse.c --- a/src/http/ngx_http_parse.c Mon Dec 23 18:11:46 2013 +0400 +++ b/src/http/ngx_http_parse.c Mon Dec 23 18:11:56 2013 +0400 @@ -1790,7 +1790,9 @@ ngx_http_parse_unsafe_uri(ngx_http_reque goto unsafe; } - if (p[0] == '.' && len == 3 && p[1] == '.' && (ngx_path_separator(p[2]))) { + if (p[0] == '.' && len > 1 && p[1] == '.' + && (len == 2 || ngx_path_separator(p[2]))) + { goto unsafe; } @@ -1816,9 +1818,11 @@ ngx_http_parse_unsafe_uri(ngx_http_reque if (ngx_path_separator(ch) && len > 2) { - /* detect "/../" */ + /* detect "/../" and "/.." */ - if (p[0] == '.' && p[1] == '.' && ngx_path_separator(p[2])) { + if (p[0] == '.' && p[1] == '.' + && (len == 3 || ngx_path_separator(p[2]))) + { goto unsafe; } } From ru at nginx.com Mon Dec 23 14:12:44 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Mon, 23 Dec 2013 14:12:44 +0000 Subject: [nginx] Teach ngx_http_parse_unsafe_uri() how to unescape URIs. Message-ID: details: http://hg.nginx.org/nginx/rev/74bfa803a5aa branches: changeset: 5491:74bfa803a5aa user: Ruslan Ermilov date: Mon Dec 23 18:12:00 2013 +0400 description: Teach ngx_http_parse_unsafe_uri() how to unescape URIs. This fixes handling of escaped URIs in X-Accel-Redirect (ticket #316), SSI (ticket #240), and DAV. diffstat: src/http/modules/ngx_http_ssi_filter_module.c | 14 ------ src/http/ngx_http_parse.c | 63 +++++++++++++++++++++++++- 2 files changed, 60 insertions(+), 17 deletions(-) diffs (128 lines): diff -r b141a7627ac6 -r 74bfa803a5aa src/http/modules/ngx_http_ssi_filter_module.c --- a/src/http/modules/ngx_http_ssi_filter_module.c Mon Dec 23 18:11:56 2013 +0400 +++ b/src/http/modules/ngx_http_ssi_filter_module.c Mon Dec 23 18:12:00 2013 +0400 @@ -1982,8 +1982,6 @@ static ngx_int_t ngx_http_ssi_include(ngx_http_request_t *r, ngx_http_ssi_ctx_t *ctx, ngx_str_t **params) { - u_char *dst, *src; - size_t len; ngx_int_t rc, key; ngx_str_t *uri, *file, *wait, *set, *stub, args; ngx_buf_t *b; @@ -2054,18 +2052,6 @@ ngx_http_ssi_include(ngx_http_request_t return rc; } - dst = uri->data; - src = uri->data; - - ngx_unescape_uri(&dst, &src, uri->len, NGX_UNESCAPE_URI); - - len = (uri->data + uri->len) - src; - if (len) { - dst = ngx_movemem(dst, src, len); - } - - uri->len = dst - uri->data; - ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "ssi include: \"%V\"", uri); diff -r b141a7627ac6 -r 74bfa803a5aa src/http/ngx_http_parse.c --- a/src/http/ngx_http_parse.c Mon Dec 23 18:11:56 2013 +0400 +++ b/src/http/ngx_http_parse.c Mon Dec 23 18:12:00 2013 +0400 @@ -1780,11 +1780,13 @@ ngx_int_t ngx_http_parse_unsafe_uri(ngx_http_request_t *r, ngx_str_t *uri, ngx_str_t *args, ngx_uint_t *flags) { - u_char ch, *p; - size_t len; + u_char ch, *p, *src, *dst; + size_t len; + ngx_uint_t quoted; len = uri->len; p = uri->data; + quoted = 0; if (len == 0 || p[0] == '?') { goto unsafe; @@ -1800,6 +1802,11 @@ ngx_http_parse_unsafe_uri(ngx_http_reque ch = *p++; + if (ch == '%') { + quoted = 1; + continue; + } + if (usual[ch >> 5] & (1 << (ch & 0x1f))) { continue; } @@ -1809,7 +1816,7 @@ ngx_http_parse_unsafe_uri(ngx_http_reque args->data = p; uri->len -= len; - return NGX_OK; + break; } if (ch == '\0') { @@ -1828,6 +1835,56 @@ ngx_http_parse_unsafe_uri(ngx_http_reque } } + if (quoted) { + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "escaped URI: \"%V\"", uri); + + src = uri->data; + + dst = ngx_pnalloc(r->pool, uri->len); + if (dst == NULL) { + return NGX_ERROR; + } + + uri->data = dst; + + ngx_unescape_uri(&dst, &src, uri->len, 0); + + uri->len = dst - uri->data; + + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "unescaped URI: \"%V\"", uri); + + len = uri->len; + p = uri->data; + + if (p[0] == '.' && len > 1 && p[1] == '.' + && (len == 2 || ngx_path_separator(p[2]))) + { + goto unsafe; + } + + for ( /* void */ ; len; len--) { + + ch = *p++; + + if (ch == '\0') { + goto unsafe; + } + + if (ngx_path_separator(ch) && len > 2) { + + /* detect "/../" and "/.." */ + + if (p[0] == '.' && p[1] == '.' + && (len == 3 || ngx_path_separator(p[2]))) + { + goto unsafe; + } + } + } + } + return NGX_OK; unsafe: From ru at nginx.com Mon Dec 23 14:12:46 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Mon, 23 Dec 2013 14:12:46 +0000 Subject: [nginx] Dav: emit a warning about unsafe URI. Message-ID: details: http://hg.nginx.org/nginx/rev/5c52ff68f380 branches: changeset: 5492:5c52ff68f380 user: Ruslan Ermilov date: Mon Dec 23 18:12:03 2013 +0400 description: Dav: emit a warning about unsafe URI. diffstat: src/http/modules/ngx_http_dav_module.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r 74bfa803a5aa -r 5c52ff68f380 src/http/modules/ngx_http_dav_module.c --- a/src/http/modules/ngx_http_dav_module.c Mon Dec 23 18:12:00 2013 +0400 +++ b/src/http/modules/ngx_http_dav_module.c Mon Dec 23 18:12:03 2013 +0400 @@ -604,7 +604,7 @@ destination_done: duri.len = last - p; duri.data = p; - flags = 0; + flags = NGX_HTTP_LOG_UNSAFE; if (ngx_http_parse_unsafe_uri(r, &duri, &args, &flags) != NGX_OK) { goto invalid_destination; From kyprizel at gmail.com Mon Dec 23 15:54:01 2013 From: kyprizel at gmail.com (kyprizel) Date: Mon, 23 Dec 2013 19:54:01 +0400 Subject: [PATCH] SSL: added support for TLS Session Tickets (RFC5077). In-Reply-To: <20131014132952.GD2144@mdounin.ru> References: <20130930142655.GD56438@mdounin.ru> <20131003151724.GI62063@mdounin.ru> <20131011142647.GQ76294@mdounin.ru> <20131014132952.GD2144@mdounin.ru> Message-ID: Do we really should fail configuration check if we were not able to read ticket key file? On Mon, Oct 14, 2013 at 5:29 PM, Maxim Dounin wrote: > Hello! > > On Fri, Oct 11, 2013 at 04:22:07PM -0700, Piotr Sikora wrote: > > > Hey Maxim, > > > > > Wouldn't it better to move ngx_ssl_session_ticket_md defines > > > to ngx_ssl_session_ticket_key_callback() implementation? > > > > You mean inside the function or just above it? I moved them just above > it. > > I'm fine with either variant. > > > > Do we really need these #ifdef's? I don't think saving several > > > bytes in ancient OpenSSL versions worth it (and the > > > ngx_ssl_stapling_index doesn't have #ifdef's). > > > > Probably not, removed. > > > > While there, I moved the definitions and initialization right after > > ngx_ssl_session_cache_index, so that the two session resumption > > indexes are together, hopefully that's alright. > > Looks good. > > > > Wouldn't ngx_ssl_ticket_key_t be better? Up to you, but > > > "ngx_ssl_session_ticket_key_t" looks a bit too long. > > > > It's a bit too long, yes, but I prefer to keep everything with the > > same prefix, so either rename everything to "ngx_ssl_ticket_" (which I > > don't like that much) or keep long names. > > I think it would be fine to name functions ngx_ssl_session_ticket_keys() > and ngx_ssl_session_ticket_key_callback(), but use shorter names for > the rest of the identifiers. Especially keeping in mind OpenSSL > calls relevant call SSL_CTX_set_tlsext_ticket_key_cb() anyway. > But I don't insist, up to you. > > > > What about something like this: > > > > > > ngx_log_debug3(NGX_LOG_DEBUG_HTTP, c->log, 0, > > > "ssl session ticket encrypt, key: \"%*s\" (%s > session)", > > > ngx_hex_dump(buf, key[0].name, 16) - buf, buf, > > > SSL_session_reused(ssl_conn) ? "reused" : > "new"); > > > > > > ? > > > > > > (Well, actually I'm not sure we need key name logged, but it > > > probably doesn't matter much.) > > > > That's much better, thanks. > > > > It's used only for the debugging, so I think it's fine to print it. > > Sometimes too much debugging is bad, too. But let's keep it as is > for now. > > > > Just in case, using a shorter name "ngx_ssl_ticket_md()" would > > > save us from wrapping here and in decrypt case. > > > > I know, I had something shorter here originally, but I've decided to > > rename it, so that it would have the same prefix... But if you really > > want to get rid of the wrap, maybe we could use "ngx_ssl_md()" here? > > What do you think? > > The "ngx_ssl_md()" looks wrong for me, as it's a message digest > used for session tickets, not something global for our ssl > implementation. > > > > Shouldn't it be ngx_memcmp(), as it checks binary data now? > > > > Good catch, thanks! That was leftover from my previous implementation. > > > > > Adding "goto found" instead of "break" would allow to avoid this > > > additional check. Up to you though. > > > > Personally, I prefer the explicit "end-of-the-loop" check, but I don't > > mind either way. Changed to "goto found". > > Minor nit: labels should be indented to match blocks, not > unindented. E.g., > > if (foo) { > if (bar) { > goto found; > } > > found: > > ... > } > > > Updated patch attached, with slightly changed commit message (better > > file names in the example). > > > > Please note, that it doesn't have any session timeout logic, so it > > should be committed after the timeout patch :) > > Sure. > > > # HG changeset patch > > # User Piotr Sikora > > # Date 1381532724 25200 > > # Fri Oct 11 16:05:24 2013 -0700 > > # Node ID 296805806a43f4b222c4cf34dd6489b37394e315 > > # Parent 5483d9e77b3287b00b1104a07688bda37bc7351e > > SSL: added ability to set keys used for Session Tickets (RFC5077). > > > > In order to support key rollover, ssl_session_ticket_key can be defined > > multiple times. The first key will be used to issue and resume Session > > Tickets, while the rest will be used only to resume them. > > > > ssl_session_ticket_key session_tickets/current.key; > > ssl_session_ticket_key session_tickets/prev-1h.key; > > ssl_session_ticket_key session_tickets/prev-2h.key; > > > > Please note that nginx supports Session Tickets even without explicit > > configuration of the keys and this feature should be only used in setups > > where SSL traffic is distributed across multiple nginx servers. > > > > Signed-off-by: Piotr Sikora > > [...] > > Committed with the followin minor tweaks: > > --- a/src/event/ngx_event_openssl.c > +++ b/src/event/ngx_event_openssl.c > @@ -2415,8 +2415,8 @@ ngx_ssl_session_ticket_key_callback(ngx_ > > RAND_pseudo_bytes(iv, 16); > EVP_EncryptInit_ex(ectx, EVP_aes_128_cbc(), NULL, key[0].aes_key, > iv); > - HMAC_Init_ex(hctx, key[0].hmac_key, 16, > ngx_ssl_session_ticket_md(), > - NULL); > + HMAC_Init_ex(hctx, key[0].hmac_key, 16, > + ngx_ssl_session_ticket_md(), NULL); > memcpy(name, key[0].name, 16); > > return 0; > @@ -2436,15 +2436,15 @@ ngx_ssl_session_ticket_key_callback(ngx_ > > return 0; > > -found: > + found: > > ngx_log_debug3(NGX_LOG_DEBUG_HTTP, c->log, 0, > "ssl session ticket decrypt, key: \"%*s\"%s", > ngx_hex_dump(buf, key[i].name, 16) - buf, buf, > (i == 0) ? " (default)" : ""); > > - HMAC_Init_ex(hctx, key[i].hmac_key, 16, > ngx_ssl_session_ticket_md(), > - NULL); > + HMAC_Init_ex(hctx, key[i].hmac_key, 16, > + ngx_ssl_session_ticket_md(), NULL); > EVP_DecryptInit_ex(ectx, EVP_aes_128_cbc(), NULL, key[i].aes_key, > iv); > > return (i == 0) ? 1 : 2 /* renew */; > > Thanks! > > -- > Maxim Dounin > http://nginx.org/en/donation.html > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Mon Dec 23 17:14:50 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 23 Dec 2013 21:14:50 +0400 Subject: [PATCH] SSL: added support for TLS Session Tickets (RFC5077). In-Reply-To: References: <20130930142655.GD56438@mdounin.ru> <20131003151724.GI62063@mdounin.ru> <20131011142647.GQ76294@mdounin.ru> <20131014132952.GD2144@mdounin.ru> Message-ID: <20131223171450.GF95113@mdounin.ru> Hello! On Mon, Dec 23, 2013 at 07:54:01PM +0400, kyprizel wrote: > Do we really should fail configuration check if we were not able to read > ticket key file? That's what configuration check is for, no? -- Maxim Dounin http://nginx.org/ From kyprizel at gmail.com Mon Dec 23 20:05:31 2013 From: kyprizel at gmail.com (kyprizel) Date: Tue, 24 Dec 2013 00:05:31 +0400 Subject: [PATCH] SSL: added support for TLS Session Tickets (RFC5077). In-Reply-To: <20131223171450.GF95113@mdounin.ru> References: <20130930142655.GD56438@mdounin.ru> <20131003151724.GI62063@mdounin.ru> <20131011142647.GQ76294@mdounin.ru> <20131014132952.GD2144@mdounin.ru> <20131223171450.GF95113@mdounin.ru> Message-ID: I mean, if something goes wrong while ticket file copying - nginx still should function, no b/c it's not essential thing? On Mon, Dec 23, 2013 at 9:14 PM, Maxim Dounin wrote: > Hello! > > On Mon, Dec 23, 2013 at 07:54:01PM +0400, kyprizel wrote: > > > Do we really should fail configuration check if we were not able to read > > ticket key file? > > That's what configuration check is for, no? > > -- > Maxim Dounin > http://nginx.org/ > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vbart at nginx.com Mon Dec 23 20:33:00 2013 From: vbart at nginx.com (Valentin V. Bartenev) Date: Tue, 24 Dec 2013 00:33 +0400 Subject: [PATCH] SSL: added support for TLS Session Tickets (RFC5077). In-Reply-To: References: <20131223171450.GF95113@mdounin.ru> Message-ID: <4420091.yHju0Mc0GF@vbart-laptop> On Tuesday 24 December 2013 00:05:31 kyprizel wrote: > I mean, if something goes wrong while ticket file copying - nginx still > should function, no b/c it's not essential thing? > And it will function. Just a quote from: http://nginx.org/en/docs/control.html "The master process first checks the syntax validity, then tries to apply new configuration, that is, to open log files and new listen sockets. If this fails, it rolls back changes and continues to work with old configuration." wbr, Valentin V. Bartenev > On Mon, Dec 23, 2013 at 9:14 PM, Maxim Dounin wrote: > > Hello! > > > > On Mon, Dec 23, 2013 at 07:54:01PM +0400, kyprizel wrote: > > > Do we really should fail configuration check if we were not able to read > > > ticket key file? > > > > That's what configuration check is for, no? > > > > -- > > Maxim Dounin > > http://nginx.org/ > > > > _______________________________________________ > > nginx-devel mailing list > > nginx-devel at nginx.org > > http://mailman.nginx.org/mailman/listinfo/nginx-devel From maxim at nginx.com Wed Dec 25 09:54:06 2013 From: maxim at nginx.com (Maxim Konovalov) Date: Wed, 25 Dec 2013 13:54:06 +0400 Subject: [PATCH] SPDY: support for SPDY v3 In-Reply-To: References: Message-ID: <52BAAB3E.4060507@nginx.com> Hi! On 12/22/13 4:56 PM, Weibin Yao wrote: > Hi, > > We have just implemented the support for SPDY v3 in nginx, with flow > control (upload and download), and a switch option between SPDY v2 and > SPDY v3. > > This patch is for Nginx-1.5.8. [...] Thanks for the submission! We already have spdy/3.1 support in our roadmap (http://trac.nginx.org/nginx/roadmap) and planning to finish this project by the end of January, 2014. During this work we will evaluate the Taobao patch too. Thanks again, Maxim Konovalov -- Maxim Konovalov http://nginx.com From dermiste at kilob.yt Wed Dec 25 12:14:57 2013 From: dermiste at kilob.yt (Vincent Gross) Date: Wed, 25 Dec 2013 23:14:57 +1100 Subject: subrequest in memory for non-upstream requests Message-ID: <20131225121457.GA21823@curry.kilob.yt> Hi folks, After some trial and errors I have a working prototype of arbitrary external http subrequests, i.e. I can fetch an external resource from a URL provided at request-time. Basically I spawn an in-memory subrequest and call ngx_http_upstream_init right after. subrequest_in_memory only works when there is an upstream. So, if the resource to be fetched is on the same server, I have 2 options: 1) establish an upstream connection to myself or 2) use an http output filter to catch the subrequest's response body. To give more context: These requests are used to fetch public key informations to authenticate http clients, so the contents will not change often, and it will have a high temporal locality. So basically we fetch the content the first time, and then we use the content in cache. Also, the odds that a key is on a foreign server far outranks the odds that the needed key is on the local server. What is the best option here ? 1), which is simple and has a low footprint, but will occasionally use an unnecessary loopback connection; or 2), which introduce no loopback connections but uses an extra http output filter to intercept the outbound bytes? Best regards, -- Vincent -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available URL: From vbart at nginx.com Thu Dec 26 13:05:22 2013 From: vbart at nginx.com (Valentin Bartenev) Date: Thu, 26 Dec 2013 13:05:22 +0000 Subject: [nginx] SPDY: fixed possible request hang. Message-ID: details: http://hg.nginx.org/nginx/rev/916cb6d28f6a branches: changeset: 5493:916cb6d28f6a user: Valentin Bartenev date: Thu Dec 26 17:03:16 2013 +0400 description: SPDY: fixed possible request hang. Processing events from upstream connection can result in sending queued frames from other streams. In this case such streams were not added to handling queue and properly handled. A global per connection flag was replaced by a per stream flag that indicates currently sending stream while all other streams can be added to handling queue. diffstat: src/http/ngx_http_spdy.c | 17 +++++++++++++---- src/http/ngx_http_spdy.h | 3 ++- src/http/ngx_http_spdy_filter_module.c | 14 ++++++++------ 3 files changed, 23 insertions(+), 11 deletions(-) diffs (110 lines): diff -r 5c52ff68f380 -r 916cb6d28f6a src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Mon Dec 23 18:12:03 2013 +0400 +++ b/src/http/ngx_http_spdy.c Thu Dec 26 17:03:16 2013 +0400 @@ -411,7 +411,7 @@ ngx_http_spdy_write_handler(ngx_event_t ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "spdy write handler"); - sc->blocked = 2; + sc->blocked = 1; rc = ngx_http_spdy_send_output_queue(sc); @@ -430,8 +430,6 @@ ngx_http_spdy_write_handler(ngx_event_t sc->last_stream = NULL; - sc->blocked = 1; - for ( /* void */ ; stream; stream = sn) { sn = stream->next; stream->handled = 0; @@ -2658,6 +2656,15 @@ ngx_http_spdy_close_stream(ngx_http_spdy } } + if (stream->handled) { + for (s = sc->last_stream; s; s = s->next) { + if (s->next == stream) { + s->next = stream->next; + break; + } + } + } + sscf = ngx_http_get_module_srv_conf(sc->http_connection->conf_ctx, ngx_http_spdy_module); @@ -2847,9 +2854,11 @@ ngx_http_spdy_finalize_connection(ngx_ht stream = sc->streams_index[i]; while (stream) { + stream->handled = 0; + r = stream->request; - fc = r->connection; + fc->error = 1; if (stream->waiting) { diff -r 5c52ff68f380 -r 916cb6d28f6a src/http/ngx_http_spdy.h --- a/src/http/ngx_http_spdy.h Mon Dec 23 18:12:03 2013 +0400 +++ b/src/http/ngx_http_spdy.h Thu Dec 26 17:03:16 2013 +0400 @@ -106,7 +106,7 @@ struct ngx_http_spdy_connection_s { ngx_uint_t last_sid; - unsigned blocked:2; + unsigned blocked:1; unsigned waiting:1; /* FIXME better name */ }; @@ -125,6 +125,7 @@ struct ngx_http_spdy_stream_s { unsigned priority:2; unsigned handled:1; + unsigned blocked:1; unsigned in_closed:1; unsigned out_closed:1; unsigned skip_data:2; diff -r 5c52ff68f380 -r 916cb6d28f6a src/http/ngx_http_spdy_filter_module.c --- a/src/http/ngx_http_spdy_filter_module.c Mon Dec 23 18:12:03 2013 +0400 +++ b/src/http/ngx_http_spdy_filter_module.c Thu Dec 26 17:03:16 2013 +0400 @@ -795,11 +795,15 @@ ngx_http_spdy_filter_get_data_frame(ngx_ static ngx_inline ngx_int_t ngx_http_spdy_filter_send(ngx_connection_t *fc, ngx_http_spdy_stream_t *stream) { + stream->blocked = 1; + if (ngx_http_spdy_send_output_queue(stream->connection) == NGX_ERROR) { fc->error = 1; return NGX_ERROR; } + stream->blocked = 0; + if (stream->waiting) { fc->buffered |= NGX_SPDY_WRITE_BUFFERED; fc->write->delayed = 1; @@ -946,16 +950,14 @@ ngx_http_spdy_handle_stream(ngx_http_spd fc->write->delayed = 0; - if (stream->handled) { + if (stream->handled || stream->blocked) { return; } - if (sc->blocked == 2) { - stream->handled = 1; + stream->handled = 1; - stream->next = sc->last_stream; - sc->last_stream = stream; - } + stream->next = sc->last_stream; + sc->last_stream = stream; } From vbart at nginx.com Thu Dec 26 13:05:24 2013 From: vbart at nginx.com (Valentin Bartenev) Date: Thu, 26 Dec 2013 13:05:24 +0000 Subject: [nginx] SPDY: refactored loop in ngx_http_spdy_body_filter(). Message-ID: details: http://hg.nginx.org/nginx/rev/27f9d5f68c1c branches: changeset: 5494:27f9d5f68c1c user: Valentin Bartenev date: Thu Dec 26 17:03:16 2013 +0400 description: SPDY: refactored loop in ngx_http_spdy_body_filter(). No functional changes. diffstat: src/http/ngx_http_spdy_filter_module.c | 14 +++++--------- 1 files changed, 5 insertions(+), 9 deletions(-) diffs (40 lines): diff -r 916cb6d28f6a -r 27f9d5f68c1c src/http/ngx_http_spdy_filter_module.c --- a/src/http/ngx_http_spdy_filter_module.c Thu Dec 26 17:03:16 2013 +0400 +++ b/src/http/ngx_http_spdy_filter_module.c Thu Dec 26 17:03:16 2013 +0400 @@ -618,7 +618,7 @@ ngx_http_spdy_body_filter(ngx_http_reque { off_t size; ngx_buf_t *b; - ngx_chain_t *cl, *ll, *out, **ln; + ngx_chain_t *cl, *out, **ln; ngx_http_spdy_stream_t *stream; ngx_http_spdy_out_frame_t *frame; @@ -644,10 +644,9 @@ ngx_http_spdy_body_filter(ngx_http_reque size = 0; ln = &out; - ll = in; - for ( ;; ) { - b = ll->buf; + do { + b = in->buf; #if 1 if (ngx_buf_size(b) == 0 && !ngx_buf_special(b)) { ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0, @@ -678,12 +677,9 @@ ngx_http_spdy_body_filter(ngx_http_reque *ln = cl; ln = &cl->next; - if (ll->next == NULL) { - break; - } + in = in->next; - ll = ll->next; - } + } while (in); if (size > NGX_SPDY_MAX_FRAME_SIZE) { ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0, From vbart at nginx.com Thu Dec 26 13:05:25 2013 From: vbart at nginx.com (Valentin Bartenev) Date: Thu, 26 Dec 2013 13:05:25 +0000 Subject: [nginx] SPDY: a bit smarter ngx_http_spdy_filter_get_data_frame(). Message-ID: details: http://hg.nginx.org/nginx/rev/97f6cd787766 branches: changeset: 5495:97f6cd787766 user: Valentin Bartenev date: Thu Dec 26 17:03:16 2013 +0400 description: SPDY: a bit smarter ngx_http_spdy_filter_get_data_frame(). There is no need to pass FLAG_FIN as a separate argument since it can always be detected from the last_buf flag of the last frame buffer. No functional changes. diffstat: src/http/ngx_http_spdy_filter_module.c | 20 ++++++++++---------- 1 files changed, 10 insertions(+), 10 deletions(-) diffs (60 lines): diff -r 27f9d5f68c1c -r 97f6cd787766 src/http/ngx_http_spdy_filter_module.c --- a/src/http/ngx_http_spdy_filter_module.c Thu Dec 26 17:03:16 2013 +0400 +++ b/src/http/ngx_http_spdy_filter_module.c Thu Dec 26 17:03:16 2013 +0400 @@ -33,8 +33,8 @@ static ngx_inline ngx_int_t ngx_http_spd ngx_connection_t *fc, ngx_http_spdy_stream_t *stream); static ngx_http_spdy_out_frame_t *ngx_http_spdy_filter_get_data_frame( - ngx_http_spdy_stream_t *stream, size_t len, ngx_uint_t flags, - ngx_chain_t *first, ngx_chain_t *last); + ngx_http_spdy_stream_t *stream, size_t len, ngx_chain_t *first, + ngx_chain_t *last); static ngx_int_t ngx_http_spdy_syn_frame_handler( ngx_http_spdy_connection_t *sc, ngx_http_spdy_out_frame_t *frame); @@ -688,7 +688,7 @@ ngx_http_spdy_body_filter(ngx_http_reque } frame = ngx_http_spdy_filter_get_data_frame(stream, (size_t) size, - b->last_buf, out, cl); + out, cl); if (frame == NULL) { return NGX_ERROR; } @@ -705,7 +705,7 @@ ngx_http_spdy_body_filter(ngx_http_reque static ngx_http_spdy_out_frame_t * ngx_http_spdy_filter_get_data_frame(ngx_http_spdy_stream_t *stream, - size_t len, ngx_uint_t fin, ngx_chain_t *first, ngx_chain_t *last) + size_t len, ngx_chain_t *first, ngx_chain_t *last) { u_char *p; ngx_buf_t *buf; @@ -727,13 +727,13 @@ ngx_http_spdy_filter_get_data_frame(ngx_ } } + flags = last->buf->last_buf ? NGX_SPDY_FLAG_FIN : 0; + ngx_log_debug4(NGX_LOG_DEBUG_HTTP, stream->request->connection->log, 0, - "spdy:%ui create DATA frame %p: len:%uz fin:%ui", - stream->id, frame, len, fin); + "spdy:%ui create DATA frame %p: len:%uz flags:%ui", + stream->id, frame, len, flags); - if (len || fin) { - - flags = fin ? NGX_SPDY_FLAG_FIN : 0; + if (len || flags) { cl = ngx_chain_get_free_buf(stream->request->pool, &stream->free_data_headers); @@ -782,7 +782,7 @@ ngx_http_spdy_filter_get_data_frame(ngx_ frame->size = NGX_SPDY_FRAME_HEADER_SIZE + len; frame->priority = stream->priority; frame->blocked = 0; - frame->fin = fin; + frame->fin = last->buf->last_buf; return frame; } From wandenberg at gmail.com Fri Dec 27 00:12:35 2013 From: wandenberg at gmail.com (Wandenberg Peixoto) Date: Thu, 26 Dec 2013 22:12:35 -0200 Subject: Help with shared memory usage In-Reply-To: <20131220164923.GK95113@mdounin.ru> References: <20130701113629.GO20717@mdounin.ru> <20130729171109.GA2130@mdounin.ru> <20130730100931.GD2130@mdounin.ru> <20131006093708.GY62063@mdounin.ru> <20131220164923.GK95113@mdounin.ru> Message-ID: Hello Maxim, I changed the patch to check only the p->next pointer. And checking if the page is in an address less than the (pool->pages + pages). + ngx_slab_page_t *prev, *p; + ngx_uint_t pages; + size_t size; + + size = pool->end - (u_char *) pool - sizeof(ngx_slab_pool_t); + pages = (ngx_uint_t) (size / (ngx_pagesize + sizeof(ngx_slab_page_t))); + + p = &page[page->slab]; + + if ((p < pool->pages + pages) && + (p->next != NULL) && + ((p->prev & NGX_SLAB_PAGE_MASK) == NGX_SLAB_PAGE)) { I hope that now I did the right checks. Regards, Wandenberg On Fri, Dec 20, 2013 at 2:49 PM, Maxim Dounin wrote: > Hello! > > On Tue, Dec 17, 2013 at 09:05:16PM -0200, Wandenberg Peixoto wrote: > > > Hi Maxim, > > > > sorry for the long delay. I hope you remember my issue. > > In attach is the new patch with the changes you suggest. > > Can you check it again? I hope it can be applied to nginx code now. > > > > About this point "2. There is probably no need to check both prev and > > next.", > > I check both pointers to avoid a segmentation fault, since in some > > situations the next can be NULL and the prev may point to pool->free. > > As far as I could follow the code seems to me that could happen one of > this > > pointers, next or prev, point to a NULL. > > I just made a double check to protect. > > As far as I see, all pages in the pool->free list are expected to > have both p->prev and p->next set. And all pages with type > NGX_SLAB_PAGE will be either on the pool->free list, or will have > p->next set to NULL. > > [...] > > > > > +{ > > > > + ngx_slab_page_t *neighbour = &page[page->slab]; > > > > > > Here "neighbour" may point outside of the allocated page > > > structures if we are freeing the last page in the pool. > > It looks like you've tried to address this problem with the > following check: > > > +static ngx_int_t > > +ngx_slab_merge_pages(ngx_slab_pool_t *pool, ngx_slab_page_t *page) > > +{ > > + ngx_slab_page_t *prev, *p; > > + > > + p = &page[page->slab]; > > + if ((u_char *) p >= pool->end) { > > + return NGX_DECLINED; > > + } > > This looks wrong, as pool->end points to the end of the pool, not > the end of allocated page structures. > > -- > Maxim Dounin > http://nginx.org/ > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: unfrag_slab_memory4.patch Type: text/x-patch Size: 1825 bytes Desc: not available URL: From vbart at nginx.com Fri Dec 27 14:48:25 2013 From: vbart at nginx.com (Valentin Bartenev) Date: Fri, 27 Dec 2013 14:48:25 +0000 Subject: [nginx] Style: removed surplus semicolons. Message-ID: details: http://hg.nginx.org/nginx/rev/9d056f10fb99 branches: changeset: 5496:9d056f10fb99 user: Valentin Bartenev date: Fri Dec 27 18:47:42 2013 +0400 description: Style: removed surplus semicolons. diffstat: src/http/modules/ngx_http_mp4_module.c | 2 +- src/http/modules/ngx_http_scgi_module.c | 2 +- src/http/modules/ngx_http_uwsgi_module.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diffs (36 lines): diff -r 97f6cd787766 -r 9d056f10fb99 src/http/modules/ngx_http_mp4_module.c --- a/src/http/modules/ngx_http_mp4_module.c Thu Dec 26 17:03:16 2013 +0400 +++ b/src/http/modules/ngx_http_mp4_module.c Fri Dec 27 18:47:42 2013 +0400 @@ -542,7 +542,7 @@ ngx_http_mp4_handler(ngx_http_request_t mp4->file.fd = of.fd; mp4->file.name = path; - mp4->file.log = r->connection->log;; + mp4->file.log = r->connection->log; mp4->end = of.size; mp4->start = (ngx_uint_t) start; mp4->request = r; diff -r 97f6cd787766 -r 9d056f10fb99 src/http/modules/ngx_http_scgi_module.c --- a/src/http/modules/ngx_http_scgi_module.c Thu Dec 26 17:03:16 2013 +0400 +++ b/src/http/modules/ngx_http_scgi_module.c Fri Dec 27 18:47:42 2013 +0400 @@ -1734,7 +1734,7 @@ ngx_http_scgi_store(ngx_conf_t *cf, ngx_ sc.source = &value[1]; sc.lengths = &scf->upstream.store_lengths; sc.values = &scf->upstream.store_values; - sc.variables = ngx_http_script_variables_count(&value[1]);; + sc.variables = ngx_http_script_variables_count(&value[1]); sc.complete_lengths = 1; sc.complete_values = 1; diff -r 97f6cd787766 -r 9d056f10fb99 src/http/modules/ngx_http_uwsgi_module.c --- a/src/http/modules/ngx_http_uwsgi_module.c Thu Dec 26 17:03:16 2013 +0400 +++ b/src/http/modules/ngx_http_uwsgi_module.c Fri Dec 27 18:47:42 2013 +0400 @@ -1916,7 +1916,7 @@ ngx_http_uwsgi_store(ngx_conf_t *cf, ngx sc.source = &value[1]; sc.lengths = &uwcf->upstream.store_lengths; sc.values = &uwcf->upstream.store_values; - sc.variables = ngx_http_script_variables_count(&value[1]);; + sc.variables = ngx_http_script_variables_count(&value[1]); sc.complete_lengths = 1; sc.complete_values = 1; From ru at nginx.com Fri Dec 27 15:41:13 2013 From: ru at nginx.com (Ruslan Ermilov) Date: Fri, 27 Dec 2013 15:41:13 +0000 Subject: [nginx] Fixed setting of content type in some cases. Message-ID: details: http://hg.nginx.org/nginx/rev/2cfc095a607a branches: changeset: 5497:2cfc095a607a user: Ruslan Ermilov date: Fri Dec 27 19:40:04 2013 +0400 description: Fixed setting of content type in some cases. This fixes content type set in stub_status and autoindex responses to be usable in content type checks made by filter modules, such as charset and sub filters. diffstat: src/http/modules/ngx_http_autoindex_module.c | 1 + src/http/modules/ngx_http_image_filter_module.c | 1 + src/http/modules/ngx_http_stub_status_module.c | 2 ++ 3 files changed, 4 insertions(+), 0 deletions(-) diffs (35 lines): diff -r 9d056f10fb99 -r 2cfc095a607a src/http/modules/ngx_http_autoindex_module.c --- a/src/http/modules/ngx_http_autoindex_module.c Fri Dec 27 18:47:42 2013 +0400 +++ b/src/http/modules/ngx_http_autoindex_module.c Fri Dec 27 19:40:04 2013 +0400 @@ -233,6 +233,7 @@ ngx_http_autoindex_handler(ngx_http_requ r->headers_out.status = NGX_HTTP_OK; r->headers_out.content_type_len = sizeof("text/html") - 1; ngx_str_set(&r->headers_out.content_type, "text/html"); + r->headers_out.content_type_lowcase = NULL; rc = ngx_http_send_header(r); diff -r 9d056f10fb99 -r 2cfc095a607a src/http/modules/ngx_http_image_filter_module.c --- a/src/http/modules/ngx_http_image_filter_module.c Fri Dec 27 18:47:42 2013 +0400 +++ b/src/http/modules/ngx_http_image_filter_module.c Fri Dec 27 19:40:04 2013 +0400 @@ -572,6 +572,7 @@ ngx_http_image_json(ngx_http_request_t * ngx_http_clean_header(r); r->headers_out.status = NGX_HTTP_OK; + r->headers_out.content_type_len = sizeof("application/json") - 1; ngx_str_set(&r->headers_out.content_type, "application/json"); r->headers_out.content_type_lowcase = NULL; diff -r 9d056f10fb99 -r 2cfc095a607a src/http/modules/ngx_http_stub_status_module.c --- a/src/http/modules/ngx_http_stub_status_module.c Fri Dec 27 18:47:42 2013 +0400 +++ b/src/http/modules/ngx_http_stub_status_module.c Fri Dec 27 19:40:04 2013 +0400 @@ -98,7 +98,9 @@ static ngx_int_t ngx_http_status_handler return rc; } + r->headers_out.content_type_len = sizeof("text/plain") - 1; ngx_str_set(&r->headers_out.content_type, "text/plain"); + r->headers_out.content_type_lowcase = NULL; if (r->method == NGX_HTTP_HEAD) { r->headers_out.status = NGX_HTTP_OK; From turchanov at farpost.com Mon Dec 30 01:35:06 2013 From: turchanov at farpost.com (Sergei Turchanov) Date: Mon, 30 Dec 2013 12:35:06 +1100 Subject: [patch] Logging: raise severity level for logged messages Message-ID: <52C0CDCA.7030508@farpost.com> # HG changeset patch # User Sergei Turchanov # Date 1388366954 -39600 # Node ID 309f46326753b70222ed86d0c09cb1f702aeffda # Parent 2cfc095a607a2e6205e45ee0be062f57a8d7ac88 Logging: raise severity level for logged messages Nginx is inconsistent in respect to severity level used for messages that are logged for user/server errors resulting in 4xx/5xx response codes. There are some places where INFO level is used which makes hard to debug user problems because setting logging level of 'error_log' to 'info' produces too much of an output whereas with this patch setting level to 'warn' gives more relevant messages. diff -r 2cfc095a607a -r 309f46326753 src/http/ngx_http_request.c --- a/src/http/ngx_http_request.c Fri Dec 27 19:40:04 2013 +0400 +++ b/src/http/ngx_http_request.c Mon Dec 30 12:29:14 2013 +1100 @@ -918,7 +918,7 @@ rc = ngx_http_validate_host(&host, r->pool, 0); if (rc == NGX_DECLINED) { - ngx_log_error(NGX_LOG_INFO, c->log, 0, + ngx_log_error(NGX_LOG_ERR, c->log, 0, "client sent invalid host in request line"); ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST); return; @@ -970,7 +970,7 @@ /* there was error while a request line parsing */ - ngx_log_error(NGX_LOG_INFO, c->log, 0, + ngx_log_error(NGX_LOG_ERR, c->log, 0, ngx_http_client_errors[rc - NGX_HTTP_CLIENT_ERROR]); ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST); return; @@ -991,7 +991,7 @@ r->request_line.len = r->header_in->end - r->request_start; r->request_line.data = r->request_start; - ngx_log_error(NGX_LOG_INFO, c->log, 0, + ngx_log_error(NGX_LOG_ERR, c->log, 0, "client sent too long URI"); ngx_http_finalize_request(r, NGX_HTTP_REQUEST_URI_TOO_LARGE); return; @@ -1023,7 +1023,7 @@ cscf = ngx_http_get_module_srv_conf(r, ngx_http_core_module); if (ngx_http_parse_complex_uri(r, cscf->merge_slashes) != NGX_OK) { - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, "client sent invalid request"); ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST); return NGX_ERROR; @@ -1070,7 +1070,7 @@ */ if (p < last && *p == '$') { - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, "client sent unsafe win32 URI"); ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST); return NGX_ERROR; @@ -1166,7 +1166,7 @@ r->lingering_close = 1; if (p == NULL) { - ngx_log_error(NGX_LOG_INFO, c->log, 0, + ngx_log_error(NGX_LOG_ERR, c->log, 0, "client sent too large request"); ngx_http_finalize_request(r, NGX_HTTP_REQUEST_HEADER_TOO_LARGE); @@ -1180,7 +1180,7 @@ p[len++] = '.'; p[len++] = '.'; p[len++] = '.'; } - ngx_log_error(NGX_LOG_INFO, c->log, 0, + ngx_log_error(NGX_LOG_ERR, c->log, 0, "client sent too long header line: \"%*s\"", len, r->header_name_start); @@ -1211,7 +1211,7 @@ /* there was error while a header line parsing */ - ngx_log_error(NGX_LOG_INFO, c->log, 0, + ngx_log_error(NGX_LOG_WARN, c->log, 0, "client sent invalid header line: \"%*s\"", r->header_end - r->header_name_start, r->header_name_start); @@ -1534,7 +1534,7 @@ return NGX_OK; } - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, "client sent duplicate header line: \"%V: %V\", " "previous value: \"%V: %V\"", &h->key, &h->value, &(*ph)->key, &(*ph)->value); @@ -1561,7 +1561,7 @@ rc = ngx_http_validate_host(&host, r->pool, 0); if (rc == NGX_DECLINED) { - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, "client sent invalid host header"); ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST); return NGX_ERROR; @@ -1714,7 +1714,7 @@ } if (r->headers_in.host == NULL && r->http_version > NGX_HTTP_VERSION_10) { - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, "client sent HTTP/1.1 request without \"Host\" header"); ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST); return NGX_ERROR; @@ -1726,7 +1726,7 @@ r->headers_in.content_length->value.len); if (r->headers_in.content_length_n == NGX_ERROR) { - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, "client sent invalid \"Content-Length\" header"); ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST); return NGX_ERROR; @@ -1734,7 +1734,7 @@ } if (r->method & NGX_HTTP_TRACE) { - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, "client sent TRACE method"); ngx_http_finalize_request(r, NGX_HTTP_NOT_ALLOWED); return NGX_ERROR; @@ -1753,7 +1753,7 @@ || ngx_strncasecmp(r->headers_in.transfer_encoding->value.data, (u_char *) "identity", 8) != 0) { - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, "client sent unknown \"Transfer-Encoding\": \"%V\"", &r->headers_in.transfer_encoding->value); ngx_http_finalize_request(r, NGX_HTTP_NOT_IMPLEMENTED); @@ -1788,7 +1788,7 @@ ngx_http_ssl_srv_conf_t *sscf; if (c->ssl == NULL) { - ngx_log_error(NGX_LOG_INFO, c->log, 0, + ngx_log_error(NGX_LOG_ERR, c->log, 0, "client sent plain HTTP request to HTTPS port"); ngx_http_finalize_request(r, NGX_HTTP_TO_HTTPS); return; @@ -1802,7 +1802,7 @@ if (rc != X509_V_OK && (sscf->verify != 3 || !ngx_ssl_verify_error_optional(rc))) { - ngx_log_error(NGX_LOG_INFO, c->log, 0, + ngx_log_error(NGX_LOG_ERR, c->log, 0, "client SSL certificate verify error: (%l:%s)", rc, X509_verify_cert_error_string(rc)); @@ -1817,7 +1817,7 @@ cert = SSL_get_peer_certificate(c->ssl->connection); if (cert == NULL) { - ngx_log_error(NGX_LOG_INFO, c->log, 0, + ngx_log_error(NGX_LOG_ERR, c->log, 0, "client sent no required SSL certificate"); ngx_ssl_remove_cached_session(sscf->ssl.ctx, @@ -2000,7 +2000,7 @@ sscf = ngx_http_get_module_srv_conf(cscf->ctx, ngx_http_ssl_module); if (sscf->verify) { - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, "client attempted to request the server name " "different from that one was negotiated"); ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST); From mdounin at mdounin.ru Mon Dec 30 20:07:51 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 31 Dec 2013 00:07:51 +0400 Subject: [patch] Logging: raise severity level for logged messages In-Reply-To: <52C0CDCA.7030508@farpost.com> References: <52C0CDCA.7030508@farpost.com> Message-ID: <20131230200750.GX95113@mdounin.ru> Hello! On Mon, Dec 30, 2013 at 12:35:06PM +1100, Sergei Turchanov wrote: > # HG changeset patch > # User Sergei Turchanov > # Date 1388366954 -39600 > # Node ID 309f46326753b70222ed86d0c09cb1f702aeffda > # Parent 2cfc095a607a2e6205e45ee0be062f57a8d7ac88 > Logging: raise severity level for logged messages > > Nginx is inconsistent in respect to severity level used for messages that > are logged for user/server errors resulting in 4xx/5xx response codes. > There are some places where INFO level is used which makes hard to debug > user problems because setting logging level of 'error_log' to 'info' > produces > too much of an output whereas with this patch setting level to 'warn' gives > more relevant messages. No, thank you. These messages are logged at info level as they are trivial to trigger by a client, and there should be an easy way to hide these messages. -- Maxim Dounin http://nginx.org/ From turchanov at farpost.com Mon Dec 30 23:10:32 2013 From: turchanov at farpost.com (Sergei Turchanov) Date: Tue, 31 Dec 2013 10:10:32 +1100 Subject: [patch] Logging: raise severity level for logged messages In-Reply-To: <20131230200750.GX95113@mdounin.ru> References: <52C0CDCA.7030508@farpost.com> <20131230200750.GX95113@mdounin.ru> Message-ID: <52C1FD68.20100@farpost.com> Okay, maybe. But what about at least this part: @@ -1801,7 +1801,7 @@ ngx_http_process_request(ngx_http_reques if (rc != X509_V_OK && (sscf->verify != 3 || !ngx_ssl_verify_error_optional(rc))) { - ngx_log_error(NGX_LOG_INFO, c->log, 0, + ngx_log_error(NGX_LOG_ERR, c->log, 0, "client SSL certificate verify error: (%l:%s)", rc, X509_verify_cert_error_string(rc)); @@ -1816,7 +1816,7 @@ ngx_http_process_request(ngx_http_reques cert = SSL_get_peer_certificate(c->ssl->connection); if (cert == NULL) { - ngx_log_error(NGX_LOG_INFO, c->log, 0, + ngx_log_error(NGX_LOG_ERR, c->log, 0, "client sent no required SSL certificate"); ngx_ssl_remove_cached_session(sscf->ssl.ctx, We use client certificate s and having ability to see client ssl certificates errors without setting error_log to 'info' level helps a lot. And these error are not just as easily triggered as plain client http errors. Don't you agree? 31.12.2013 07:07, Maxim Dounin ???????: > Hello! > > On Mon, Dec 30, 2013 at 12:35:06PM +1100, Sergei Turchanov wrote: > >> # HG changeset patch >> # User Sergei Turchanov >> # Date 1388366954 -39600 >> # Node ID 309f46326753b70222ed86d0c09cb1f702aeffda >> # Parent 2cfc095a607a2e6205e45ee0be062f57a8d7ac88 >> Logging: raise severity level for logged messages >> >> Nginx is inconsistent in respect to severity level used for messages that >> are logged for user/server errors resulting in 4xx/5xx response codes. >> There are some places where INFO level is used which makes hard to debug >> user problems because setting logging level of 'error_log' to 'info' >> produces >> too much of an output whereas with this patch setting level to 'warn' gives >> more relevant messages. > > No, thank you. These messages are logged at info level as they > are trivial to trigger by a client, and there should be an easy > way to hide these messages. > From mdounin at mdounin.ru Mon Dec 30 23:37:36 2013 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 31 Dec 2013 03:37:36 +0400 Subject: [patch] Logging: raise severity level for logged messages In-Reply-To: <52C1FD68.20100@farpost.com> References: <52C0CDCA.7030508@farpost.com> <20131230200750.GX95113@mdounin.ru> <52C1FD68.20100@farpost.com> Message-ID: <20131230233736.GA95113@mdounin.ru> Hello! On Tue, Dec 31, 2013 at 10:10:32AM +1100, Sergei Turchanov wrote: > Okay, maybe. But what about at least this part: > > @@ -1801,7 +1801,7 @@ ngx_http_process_request(ngx_http_reques > if (rc != X509_V_OK > && (sscf->verify != 3 || > !ngx_ssl_verify_error_optional(rc))) > { > - ngx_log_error(NGX_LOG_INFO, c->log, 0, > + ngx_log_error(NGX_LOG_ERR, c->log, 0, > "client SSL certificate verify error: > (%l:%s)", > rc, X509_verify_cert_error_string(rc)); > > @@ -1816,7 +1816,7 @@ ngx_http_process_request(ngx_http_reques > cert = SSL_get_peer_certificate(c->ssl->connection); > > if (cert == NULL) { > - ngx_log_error(NGX_LOG_INFO, c->log, 0, > + ngx_log_error(NGX_LOG_ERR, c->log, 0, > "client sent no required SSL > certificate"); > > ngx_ssl_remove_cached_session(sscf->ssl.ctx, > > > We use client certificate s and having ability to see client ssl > certificates errors > without setting error_log to 'info' level helps a lot. And these error are > not just > as easily triggered as plain client http errors. Don't you agree? They are easily triggered by clients, even assuming non-malicious clients. Moreover, e.g., Safari on OS X used to supply arbitrary certificates in a response to a certificate request, see here: http://trac.nginx.org/nginx/ticket/472 If you want to trace such problems without using the 'info' logging level, adding the $ssl_client_verify variable to access logs might be a good idea, see here: http://nginx.org/en/docs/http/ngx_http_ssl_module.html#variables Additionally, there are special status codes available for these conditions, to facilitate automatic detection and hanlding of these errors, see here: http://nginx.org/en/docs/http/ngx_http_ssl_module.html#errors -- Maxim Dounin http://nginx.org/ From richard at fussenegger.info Tue Dec 31 12:22:59 2013 From: richard at fussenegger.info (Richard Fussenegger, BSc) Date: Tue, 31 Dec 2013 13:22:59 +0100 Subject: Syntax highlighting for nano In-Reply-To: References: Message-ID: <52C2B723.3050408@fussenegger.info> I've seen that the latest nginx release contains syntax highlighting for vim. I created a simple syntax highlighting scheme for nano some time ago. Maybe you'd like to include it as well. You can find it via the following link: https://github.com/Fleshgrinder/nano-editor-conf-syntax-highlighting This may be a catch all for conf files, but I initially created it for my nginx files and it works great. Regards Richard From richard at fussenegger.info Tue Dec 31 12:35:52 2013 From: richard at fussenegger.info (Richard Fussenegger, BSc) Date: Tue, 31 Dec 2013 13:35:52 +0100 Subject: LSB compliant init script (e.g. Debian 7) In-Reply-To: References: Message-ID: <52C2BA28.7030502@fussenegger.info> I also happen to have a LSB compliant init script for nginx. I think some lines should be removed for inclusion in the nginx source (everything that has to do with the temporary paths), but it's a rock solid starting point. https://github.com/MovLib/www/blob/master/bin/init-nginx.sh The script allows you to use "service nginx {force-reload|reload|restart|start|status|stop}" and "nginx -t" is always executed before attempting to do anything. This ensures that the service isn't interrupted because of some misconfiguration. It's also using the various LSB functions for some eye candy. Hope you like it. Regards Richard -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4264 bytes Desc: S/MIME Cryptographic Signature URL: From info at phpgangsta.de Tue Dec 31 16:38:09 2013 From: info at phpgangsta.de (Michael Kliewe) Date: Tue, 31 Dec 2013 17:38:09 +0100 Subject: Support of IMAP ID command in mail proxy module? Message-ID: <52C2F2F1.40308@phpgangsta.de> Hello, I'm using the mail module of nginx to proxy and loadbalance IMAP+POP3 connections to backend servers. I do authentication via http authentication. Some Clients are sending IMAP ID commands to the server with information about their software and version. I would like to log that and maybe use that during authentication. It would be great if nginx would support that command, and if it has been sent before LOGIN, also provide the information to the http authentication script. IMAP ID can be found in RFC 2971: http://www.faqs.org/rfcs/rfc2971.html The information about the client could be used to route the user to a specific backend server that has some client-specific IMAP bug fixes in place, or could be used to restrict logins of a specific user to only one specific client if the user wants that for slightly higher security, or for a list of "last activity of the user" like GMail does, or just for client statistics. I'm not a C programmer, so sadly I cannot write a patch myself for this, but maybe someone is able to add this small feature? Thanks Michael