From daniel.black at openquery.com Tue Apr 1 06:57:10 2014 From: daniel.black at openquery.com (Daniel Black) Date: Tue, 1 Apr 2014 16:57:10 +1000 (EST) Subject: SSL Session Caching - Memcached In-Reply-To: Message-ID: <1927727282.1566.1396335430742.JavaMail.root@zimbra.lentz.com.au> ----- Original Message ----- > Hi nginx-devel, > > > I'm new to the world of nginx development. I'm looking at patching > nginx to allow for configurable session caching via memcached. Goal is > to be able to session cache across boxes within a cluster when there's > no guarantee that a client will connect to the same box each time. > > > I've seen this previous work from mpalmer: > https://github.com/mpalmer/nginx/compare/0.8.x...memcache-ssl-0.8 > > > > However it is 3 years old. I'm going to look at patching it onto the > mainline and auditing the code to see if it still makes sense. > > > A few questions. > 1) Is there anything I should look out for / be careful of? > 2) Is there any desire for this feature on mainline? If so, any > suggestions on how to request feedback. Via patches to the mailing > list? http://trac.nginx.org/nginx/ticket/120#comment:8 (Just a side note: Matt Palmer's patches are classic example of what shouldn't be done in nginx. He uses blocking remote memcached calls, and this is something one shouldn't even think about as this will block the whole nginx worker and all clients.) On another note, there is session tickets in files now implemented that can be distributed. Sessions however are different. -- Daniel Black, Engineer @ Open Query (http://openquery.com.au) Remote expertise & maintenance for MySQL/MariaDB server environments. From ykirpichev at gmail.com Tue Apr 1 10:54:56 2014 From: ykirpichev at gmail.com (Yury Kirpichev) Date: Tue, 1 Apr 2014 14:54:56 +0400 Subject: [nginx] Vary header is repeated twice in response Message-ID: Hi, I've got an issue that "Vary" header is repeated twice in response when "gzip_vary on" is specified in config file; My configuration is the following: Two instances of nginx are running on different hosts (A and B) There is location /smth/ { proxy_pass http://B/smth; } and gzip_vary on; in config for host A. B adds "Vary: Accept-Encoding" in response for http://B/smth And then if http://A/smth request is performed "Vary" header is returned twice in response. < Connection: keep-alive < Vary: Accept-Encoding < Vary: Accept-Encoding < date: Tue, 01 Apr 2014 10:02:27 GMT < expires: Tue, 01 Apr 2014 10:07:27 GMT < server: nginx/1.4.4 Could you please help me to resolve this problem? Is it known issue or it is normal behaviour or may be something wrong on my side? Thanks, BR/ Yury -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Tue Apr 1 11:04:15 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 1 Apr 2014 15:04:15 +0400 Subject: [nginx] Vary header is repeated twice in response In-Reply-To: References: Message-ID: <20140401110415.GU34696@mdounin.ru> Hello! On Tue, Apr 01, 2014 at 02:54:56PM +0400, Yury Kirpichev wrote: > Hi, > > I've got an issue that "Vary" header is repeated twice in response when > "gzip_vary on" is specified in config file; > > My configuration is the following: > Two instances of nginx are running on different hosts (A and B) > There is > location /smth/ { > proxy_pass http://B/smth; > } > > and > gzip_vary on; > in config for host A. > > B adds "Vary: Accept-Encoding" in response for http://B/smth > > And then if http://A/smth request is performed "Vary" header is returned > twice in response. > < Connection: keep-alive > < Vary: Accept-Encoding > < Vary: Accept-Encoding > < date: Tue, 01 Apr 2014 10:02:27 GMT > < expires: Tue, 01 Apr 2014 10:07:27 GMT > < server: nginx/1.4.4 > > > Could you please help me to resolve this problem? Is it known issue or it > is normal behaviour or may be something wrong on my side? This is certainly not relevant to nginx-devel@, please use nginx@ mailing list for such questions (Cc'd). -- Maxim Dounin http://nginx.org/ From mdounin at mdounin.ru Tue Apr 1 13:50:10 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 01 Apr 2014 13:50:10 +0000 Subject: [nginx] Core: hash now ignores bucket_size if it hits max_size l... Message-ID: details: http://hg.nginx.org/nginx/rev/c348dea081fb branches: changeset: 5635:c348dea081fb user: Maxim Dounin date: Mon Mar 31 21:40:31 2014 +0400 description: Core: hash now ignores bucket_size if it hits max_size limit. diffstat: src/core/ngx_hash.c | 13 +++++-------- 1 files changed, 5 insertions(+), 8 deletions(-) diffs (24 lines): diff --git a/src/core/ngx_hash.c b/src/core/ngx_hash.c --- a/src/core/ngx_hash.c +++ b/src/core/ngx_hash.c @@ -312,15 +312,12 @@ ngx_hash_init(ngx_hash_init_t *hinit, ng continue; } - ngx_log_error(NGX_LOG_EMERG, hinit->pool->log, 0, - "could not build the %s, you should increase " - "either %s_max_size: %i or %s_bucket_size: %i", + ngx_log_error(NGX_LOG_WARN, hinit->pool->log, 0, + "could not build optimal %s, you should increase " + "either %s_max_size: %i or %s_bucket_size: %i; " + "ignoring %s_bucket_size", hinit->name, hinit->name, hinit->max_size, - hinit->name, hinit->bucket_size); - - ngx_free(test); - - return NGX_ERROR; + hinit->name, hinit->bucket_size, hinit->name); found: From mdounin at mdounin.ru Tue Apr 1 13:50:11 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 01 Apr 2014 13:50:11 +0000 Subject: [nginx] Core: fixed hash to actually try max_size. Message-ID: details: http://hg.nginx.org/nginx/rev/54f847c88cf7 branches: changeset: 5636:54f847c88cf7 user: Maxim Dounin date: Mon Mar 31 21:40:35 2014 +0400 description: Core: fixed hash to actually try max_size. Previously, maximum size of a hash table built was (max_size - 1). diffstat: src/core/ngx_hash.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff --git a/src/core/ngx_hash.c b/src/core/ngx_hash.c --- a/src/core/ngx_hash.c +++ b/src/core/ngx_hash.c @@ -282,7 +282,7 @@ ngx_hash_init(ngx_hash_init_t *hinit, ng start = hinit->max_size - 1000; } - for (size = start; size < hinit->max_size; size++) { + for (size = start; size <= hinit->max_size; size++) { ngx_memzero(test, size * sizeof(u_short)); From mdounin at mdounin.ru Tue Apr 1 13:50:13 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 01 Apr 2014 13:50:13 +0000 Subject: [nginx] Adjusted default value of types_hash_bucket_size (ticket... Message-ID: details: http://hg.nginx.org/nginx/rev/5a65b9d8bc2b branches: changeset: 5637:5a65b9d8bc2b user: Maxim Dounin date: Mon Mar 31 22:47:42 2014 +0400 description: Adjusted default value of types_hash_bucket_size (ticket #352). The ngx_cacheline_size may be too low on some platforms, resulting in unexpected hash build problems (as no collisions are tolerated due to low bucket_size, and max_size isn't big enough to build a hash without collisions). These problems aren't fatal anymore but nevertheless need to be addressed. diffstat: src/http/ngx_http_core_module.c | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diffs (13 lines): diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c +++ b/src/http/ngx_http_core_module.c @@ -3710,8 +3710,7 @@ ngx_http_core_merge_loc_conf(ngx_conf_t prev->types_hash_max_size, 1024); ngx_conf_merge_uint_value(conf->types_hash_bucket_size, - prev->types_hash_bucket_size, - ngx_cacheline_size); + prev->types_hash_bucket_size, 64); conf->types_hash_bucket_size = ngx_align(conf->types_hash_bucket_size, ngx_cacheline_size); From arut at nginx.com Tue Apr 1 16:54:28 2014 From: arut at nginx.com (Roman Arutyunyan) Date: Tue, 01 Apr 2014 16:54:28 +0000 Subject: [nginx] Mp4: allow end values bigger than track duration. Message-ID: details: http://hg.nginx.org/nginx/rev/0c0dd1aacdf5 branches: changeset: 5638:0c0dd1aacdf5 user: Roman Arutyunyan date: Tue Apr 01 20:53:18 2014 +0400 description: Mp4: allow end values bigger than track duration. If start time is within the track but end time is out of it, error "end time is out mp4 stts samples" is generated. However it's better to ignore the error and output the track until its end. diffstat: src/http/modules/ngx_http_mp4_module.c | 20 +++++++++++++++----- 1 files changed, 15 insertions(+), 5 deletions(-) diffs (30 lines): diff -r 5a65b9d8bc2b -r 0c0dd1aacdf5 src/http/modules/ngx_http_mp4_module.c --- a/src/http/modules/ngx_http_mp4_module.c Mon Mar 31 22:47:42 2014 +0400 +++ b/src/http/modules/ngx_http_mp4_module.c Tue Apr 01 20:53:18 2014 +0400 @@ -2154,11 +2154,21 @@ ngx_http_mp4_crop_stts_data(ngx_http_mp4 entry++; } - ngx_log_error(NGX_LOG_ERR, mp4->file.log, 0, - "%s time is out mp4 stts samples in \"%s\"", - start ? "start" : "end", mp4->file.name.data); - - return NGX_ERROR; + if (start) { + ngx_log_error(NGX_LOG_ERR, mp4->file.log, 0, + "start time is out mp4 stts samples in \"%s\"", + mp4->file.name.data); + + return NGX_ERROR; + + } else { + trak->end_sample = trak->start_sample + start_sample; + + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, + "end_sample:%ui", trak->end_sample); + + return NGX_OK; + } found: From mdounin at mdounin.ru Tue Apr 1 17:00:37 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 1 Apr 2014 21:00:37 +0400 Subject: [PATCH] SSL: support automatic selection of ECDH temporary key parameters In-Reply-To: References: Message-ID: <20140401170037.GZ34696@mdounin.ru> Hello! On Thu, Jan 30, 2014 at 03:23:01PM -0800, Piotr Sikora wrote: > Hello, > slightly better patch attached. > > Hopefully, now that ALPN (OpenSSL-1.0.2+ feature) is in, this can be > also committed :) > > Best regards, > Piotr Sikora > > > # HG changeset patch > # User Piotr Sikora > # Date 1391123946 28800 > # Thu Jan 30 15:19:06 2014 -0800 > # Node ID 63c549cc2e817772979238f055b72a4f96198a2a > # Parent 2e40188f83ef5bf1ae5afe0dd445689049f46a5c > SSL: support automatic selection of ECDH temporary key parameters. > > When compiled against OpenSSL-1.0.2+, the colon separated list of > supported curves can be provided using either curve NIDs: > > ssl_ecdh_curve secp521r1:secp384r1:prime256v1; > > or names: > > ssl_ecdh_curve P-521:P-384:P-256; > > Signed-off-by: Piotr Sikora > > diff -r 2e40188f83ef -r 63c549cc2e81 src/event/ngx_event_openssl.c > --- a/src/event/ngx_event_openssl.c Thu Jan 30 19:13:12 2014 +0400 > +++ b/src/event/ngx_event_openssl.c Thu Jan 30 15:19:06 2014 -0800 > @@ -681,6 +681,23 @@ ngx_ssl_ecdh_curve(ngx_conf_t *cf, ngx_s > { > #if OPENSSL_VERSION_NUMBER >= 0x0090800fL > #ifndef OPENSSL_NO_ECDH > +#ifdef SSL_CTRL_SET_ECDH_AUTO > + > + if (SSL_CTX_set1_curves_list(ssl->ctx, name->data) == 0) { > + ngx_ssl_error(NGX_LOG_EMERG, ssl->log, 0, > + "Unknown curve in \"%s\"", name->data); > + return NGX_ERROR; > + } > + > + if (SSL_CTX_set_ecdh_auto(ssl->ctx, 1) == 0) { > + ngx_ssl_error(NGX_LOG_EMERG, ssl->log, 0, > + "Unable to set automatic curve selection for \"%s\"", > + name->data); > + return NGX_ERROR; > + } > + > +#else > + > int nid; > EC_KEY *ecdh; > > @@ -710,6 +727,8 @@ ngx_ssl_ecdh_curve(ngx_conf_t *cf, ngx_s > SSL_CTX_set_tmp_ecdh(ssl->ctx, ecdh); > > EC_KEY_free(ecdh); > + > +#endif > #endif > #endif Sorry for late reponse, but I finally managed to get some time to look into the SSL_CTX_set_ecdh_auto() / SSL_CTX_set1_curves_list() support in yet-to-be-released OpenSSL 1.0.2. Comments about the patch below, in no particular order: - Suggested code doesn't seem to allow to use the default list of curves, as normally available with just a call to SSL_CTX_set_ecdh_auto(); this seems to be what OpenSSL recommends to use by default, and we may want to follow. - Error messages in the ngx_ssl_ecdh_curve() are way off from what's normally used in ngx_event_openssl.c, and probably it's not a good idea to use similar messages in the new code. - If nginx was compiled with OpenSSL 1.0.2, but used with an older version, things will not work at all; this is not something completely unacceptable, but it's something we may want to avoid. - SSL_CTX_set_options(SSL_OP_SINGLE_ECDH_USE) is not used with OpenSSL 1.0.2, and this looks just wrong. -- Maxim Dounin http://nginx.org/ From nipunn at dropbox.com Wed Apr 2 01:06:10 2014 From: nipunn at dropbox.com (Nipunn Koorapati) Date: Tue, 1 Apr 2014 18:06:10 -0700 Subject: SSL Session Caching - Memcached In-Reply-To: References: Message-ID: Sup, I was able to graft the patch. It compiles and runs successfully. It required a bit more work obviously as the code has changed since version 0.8, but I think I covered it. Also had to make some modifications to the mail-ssl module as it had dependencies. Is there some nginx tests / testsuite module I should verify against / add tests to? Thanks --Nipunn On Mon, Mar 31, 2014 at 4:26 PM, Nipunn Koorapati wrote: > Hi nginx-devel, > > I'm new to the world of nginx development. I'm looking at patching nginx > to allow for configurable session caching via memcached. Goal is to be able > to session cache across boxes within a cluster when there's no guarantee > that a client will connect to the same box each time. > > I've seen this previous work from mpalmer: > https://github.com/mpalmer/nginx/compare/0.8.x...memcache-ssl-0.8 > > However it is 3 years old. I'm going to look at patching it onto the > mainline and auditing the code to see if it still makes sense. > > A few questions. > 1) Is there anything I should look out for / be careful of? > 2) Is there any desire for this feature on mainline? If so, any > suggestions on how to request feedback. Via patches to the mailing list? > > Thanks! > --Nipunn > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Wed Apr 2 12:35:24 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 2 Apr 2014 16:35:24 +0400 Subject: SSL Session Caching - Memcached In-Reply-To: References: Message-ID: <20140402123523.GB34696@mdounin.ru> Hello! On Tue, Apr 01, 2014 at 06:06:10PM -0700, Nipunn Koorapati wrote: > I was able to graft the patch. It compiles and runs successfully. It > required a bit more work obviously as the code has changed since version > 0.8, but I think I covered it. Also had to make some modifications to the > mail-ssl module as it had dependencies. Is there some nginx tests / > testsuite module I should verify against / add tests to? There is a test suite as available at http://hg.nginx.org/nginx-tests. On the other hand, as already pointed out by Daniel, the patch in question isn't something to be seriously considered. It's just a dirty hack. -- Maxim Dounin http://nginx.org/ From nipunn at dropbox.com Wed Apr 2 21:37:28 2014 From: nipunn at dropbox.com (Nipunn Koorapati) Date: Wed, 2 Apr 2014 14:37:28 -0700 Subject: SSL Session Caching - Memcached In-Reply-To: <20140402123523.GB34696@mdounin.ru> References: <20140402123523.GB34696@mdounin.ru> Message-ID: Ah. I hadn't seen Daniel's response yet. Thanks for the reply. I looked into session tickets, but they do not provide forward secrecy. Unless keys are randomly generated and rotated with high frequency, there will exist a single point of attack on the side. The cached session-id mechanism provides this, which is why I'm looking into it. Of course, I understand your point about synchronous blocking operations. It looks like memcached provides asynchronous get/set methods, however it doesn't look like there are hooks into OpenSSL to split up the SSL_CTX_sess_set_*_cb methods. For the memcache_set call, we could switch to being asynchronous fairly easily. For the memcache_get call, memcache provides 2 separate methods. We could do some additional h4x and setjmp/longjmp from the memcache call point to our event loop, but I assume there's going to be some pushback to that idea. In either case, we'd also want aggressive configurable timeouts for the calls to memcached. Is there some philosophical qualm with adding a dependency to memcache, or is it just the concern of having a blocking call in the worker event loop. I think we may be able to work around that. Thanks --Nipunn On Wed, Apr 2, 2014 at 5:35 AM, Maxim Dounin wrote: > Hello! > > On Tue, Apr 01, 2014 at 06:06:10PM -0700, Nipunn Koorapati wrote: > > > I was able to graft the patch. It compiles and runs successfully. It > > required a bit more work obviously as the code has changed since version > > 0.8, but I think I covered it. Also had to make some modifications to the > > mail-ssl module as it had dependencies. Is there some nginx tests / > > testsuite module I should verify against / add tests to? > > There is a test suite as available at > http://hg.nginx.org/nginx-tests. > > On the other hand, as already pointed out by Daniel, the patch in > question isn't something to be seriously considered. It's just a > dirty hack. > > -- > Maxim Dounin > http://nginx.org/ > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Thu Apr 3 13:47:51 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 3 Apr 2014 17:47:51 +0400 Subject: SSL Session Caching - Memcached In-Reply-To: References: <20140402123523.GB34696@mdounin.ru> Message-ID: <20140403134751.GO34696@mdounin.ru> Hello! On Wed, Apr 02, 2014 at 02:37:28PM -0700, Nipunn Koorapati wrote: > Ah. I hadn't seen Daniel's response yet. Thanks for the reply. > > I looked into session tickets, but they do not provide forward secrecy. > Unless keys are randomly generated and rotated with high frequency, there > will exist a single point of attack on the side. The cached session-id > mechanism provides this, which is why I'm looking into it. As long as you cache sessions, forward secrecy isn't provided for still cached sessions either. Moreover, storing sessions in memcached means that they can be easily accessed from a network where the memcached server is. While this is solvable, it's hardly seriously better than shared session ticket keys with proper rotation. > Of course, I understand your point about synchronous blocking operations. > It looks like memcached provides asynchronous get/set methods, however it > doesn't look like there are hooks into OpenSSL to split up the > SSL_CTX_sess_set_*_cb methods. Sure, OpenSSL as of now doesn't provide non-blocking session retrieval callbacks. So the only sensible approach for distributed session cache is to keep all sessions on all nodes. When this was discussed last time on this mailing list, the result was ssl_session_ticket_key implementation. > For the memcache_set call, we could switch > to being asynchronous fairly easily. For the memcache_get call, memcache > provides 2 separate methods. We could do some additional h4x and > setjmp/longjmp from the memcache call point to our event loop, but I assume > there's going to be some pushback to that idea. > > In either case, we'd also want aggressive configurable timeouts for the > calls to memcached. > > Is there some philosophical qualm with adding a dependency to memcache, or > is it just the concern of having a blocking call in the worker event loop. > I think we may be able to work around that. You are free to do whatever you want. I just pointed out why this is not going to be included into nginx. > > Thanks > --Nipunn > > > > On Wed, Apr 2, 2014 at 5:35 AM, Maxim Dounin wrote: > > > Hello! > > > > On Tue, Apr 01, 2014 at 06:06:10PM -0700, Nipunn Koorapati wrote: > > > > > I was able to graft the patch. It compiles and runs successfully. It > > > required a bit more work obviously as the code has changed since version > > > 0.8, but I think I covered it. Also had to make some modifications to the > > > mail-ssl module as it had dependencies. Is there some nginx tests / > > > testsuite module I should verify against / add tests to? > > > > There is a test suite as available at > > http://hg.nginx.org/nginx-tests. > > > > On the other hand, as already pointed out by Daniel, the patch in > > question isn't something to be seriously considered. It's just a > > dirty hack. > > > > -- > > Maxim Dounin > > http://nginx.org/ > > > > _______________________________________________ > > nginx-devel mailing list > > nginx-devel at nginx.org > > http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -- Maxim Dounin http://nginx.org/ From c0nw0nk at hotmail.co.uk Fri Apr 4 19:19:29 2014 From: c0nw0nk at hotmail.co.uk (C0nw0nk W0nky) Date: Fri, 4 Apr 2014 20:19:29 +0100 Subject: Windows | Nginx Mapped Hard Drive | Network Sharing Message-ID: http://stackoverflow.com/questions/22870814/nginx-mapped-hard-drive-network-sharing So i tried sharing my hard drives on windows and serving the content on them from nginx but nginx returns a 404 not found error everytime. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Fri Apr 4 19:40:16 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 4 Apr 2014 23:40:16 +0400 Subject: Windows | Nginx Mapped Hard Drive | Network Sharing In-Reply-To: References: Message-ID: <20140404194016.GU34696@mdounin.ru> Hello! On Fri, Apr 04, 2014 at 08:19:29PM +0100, C0nw0nk W0nky wrote: > http://stackoverflow.com/questions/22870814/nginx-mapped-hard-drive-network-sharing > > So i tried sharing my hard drives on windows and serving the > content on them from nginx but nginx returns a 404 not found > error everytime. This question is not relevant to nginx-devel@ mailing list. Please use nginx@ mailing list for questions not related to nginx development. Thank you. -- Maxim Dounin http://nginx.org/ From agentzh at gmail.com Sun Apr 6 04:11:15 2014 From: agentzh at gmail.com (Yichun Zhang (agentzh)) Date: Sat, 5 Apr 2014 21:11:15 -0700 Subject: [ANN] Test::Nginx 0.23 released Message-ID: Hello! I've just uploaded Test::Nginx 0.23 to CPAN: http://search.cpan.org/perldoc?Test::Nginx It will appear on the CPAN mirror near you in the next few hours or so. Special thanks go to all our contributors and users :) Here's the complete change log for this release (compared to the last CPAN release, 0.22): * feature: added new section --- response_body_unlike. thanks Rickey Visinski for the patch. * feature: allow use of the $ServerPort and $ServerPortForClient variables in "--- response_headers". * feature: now we automatically kill the process listening on the mocked TCP port when we have tried for more than 20 times. * feature: added new section "--- http09" to Test::Nginx::Socket to indicate use of HTTP 0.9 requests. * feature: added support for the $LIBLUA_PATH variable in "--- stap", which evaluates to the absolute path of the liblua*.so DSO loaded by the nginx server. * feature: we skipped reading the response body for 101 responses. * feature: added new section "--- post_main_config" to add custom top-level configuration snippets right after the http {} block generated by Test::Nginx. * feature: added unix domain socket support for tcp_listen. thanks Jiale Zhi for the patch. * feature: added new sections "--- grep_error_log" and "--- grep_error_log_out". * feature: added utility function check_accum_error_log for checking the accumulated error logs in "--- error_log" and "--- no_error_log". * feature: added utility Perl function server_port_for_client. thanks doujiang for the patch. * feature: added configuration Perl function server_name(). thanks doujiang for the original patch. * feature: added new module Test::Nginx::Socket::Lua. * feature: automatically test if there is any assertion failures in the nginx error logs. * feature: when nginx process is gone unexpectedly, we automatically tail the nginx error log file. * feature: added the "--- no_check_leak" section to skip test blocks in the "check leak" testing mode. * feature: "--- no_error_log" accepts a single Perl Regexp object now, just like "--- error_log". * bugfix: Test::Nginx::Socket tried to read response bodies for 304 responses. * bugfix: we did not try hard enough before killing the nginx process by force. * bugfix: improved the error handling when recv() fails on the HTTP connection. * bugfix: subtests for "--- tcp_query", "--- tcp_query_len", and "--- udp_query" disabled test plan checks. * bugfix: we did not pass the LD_LIBRARY_PATH environment to nginx when systemtap is used. * bugfix: "--- error_log" should work on each test request's own logs only. * bugfix: server address should default to 127.0.0.1 instead of "localhost". * bugfix: made the test skip messages better. * change: removed "env LUA_PATH" and "env LUA_CPATH" from the default nginx configuration generated. * change: set --num-callers=100 to valgrind by default. * doc: Test::Nginx::Socket: documented the various exported Perl functions like "run_tests", "repeat_each", and "no_shuffle". This Perl module provides a test scaffold based on IO::Socket or LWP for automated testing in Nginx C module or ngx_lua-based Lua library development. This class inherits from Test::Base, thus bringing all its declarative power to the NginxC module testing practices. Please check out the full documentation on CPAN: http://search.cpan.org/perldoc?Test::Nginx::Socket All of our Nginx modules (as well as our lua-resty-* libraries) are using Test::Nginx to drive their test suites. And it is also driving my test cluster running on Amazon EC2: http://qa.openresty.org Please note that this module is completely different from the Test::Nginx module created by Maxim Dounin. The git repository for this Perl module is hosted on GitHub: https://github.com/agentzh/test-nginx Have fun! -agentzh From piotr at cloudflare.com Mon Apr 7 02:09:41 2014 From: piotr at cloudflare.com (Piotr Sikora) Date: Sun, 6 Apr 2014 19:09:41 -0700 Subject: [PATCH] SSL: support automatic selection of ECDH temporary key parameters In-Reply-To: <20140401170037.GZ34696@mdounin.ru> References: <20140401170037.GZ34696@mdounin.ru> Message-ID: Hey Maxim, > Comments about the patch below, in no particular order: > > - Suggested code doesn't seem to allow to use the default list of > curves, as normally available with just a call to > SSL_CTX_set_ecdh_auto(); this seems to be what OpenSSL > recommends to use by default, and we may want to follow. But isn't that the way nginx usually interacts with OpenSSL? i.e. always calling SSL_CTX_set_cipher_list(), even with the default "DEFAULT" value? Also, right now nginx uses NIST P-256 curve. Calling SSL_CTX_set_ecdh_auto() without limiting list of supported curves enables all of them, with preference for the strongest ones, which means that all modern browsers will start using NIST P-521 and clients compiled against OpenSSL will start using NIST B-571. This results in rather significant performance loss: op op/s 256 bit ecdh (nistp256) 0.0002s 4589.7 521 bit ecdh (nistp521) 0.0006s 1551.9 571 bit ecdh (nistb571) 0.0030s 330.9 and I don't think that it's a good idea to silently change used curve for users using default configuration. > - Error messages in the ngx_ssl_ecdh_curve() are way off from > what's normally used in ngx_event_openssl.c, and probably > it's not a good idea to use similar messages in the new code. Right, my bad. > - If nginx was compiled with OpenSSL 1.0.2, but used with an > older version, things will not work at all; this is not something > completely unacceptable, but it's something we may want to > avoid. Will look into it. > - SSL_CTX_set_options(SSL_OP_SINGLE_ECDH_USE) is not used > with OpenSSL 1.0.2, and this looks just wrong. Well, it just looks wrong ;) If you dig into OpenSSL's code, you'll notice that this option is checked only in 3 places: when setting SSL_CTX_set_tmp_ecdh() / SSL_set_tmp_ecdh() to see if we need to generate "persistent" temporary key and during ServerKeyExchange phase to see if we need to generate ephemeral temporary key. This check consists of 3 conditions and first two are fulfilled via SSL_CTX_set_ecdh_auto(), so SSL_OP_SINGLE_ECDH_USE is never check when using it. Basically, SSL_CTX_set_ecdh_auto(ssl->ctx, 1); is equivalent to: SSL_CTX_set_options(ssl->ctx, SSL_OP_SINGLE_ECDH_USE); SSL_CTX_set_tmp_ecdh(ssl->ctx, ecdh); Having said that, I don't mind re-adding it, if it's going to make you happier. Best regards, Piotr Sikora From hc0606 at gmail.com Mon Apr 7 08:34:52 2014 From: hc0606 at gmail.com (Han Cheng) Date: Mon, 7 Apr 2014 16:34:52 +0800 Subject: [PATCH] Core: configurable listening try number Message-ID: Hello, there, I'm a new guy to nginx. Recently, I'm reading the source code. I found some TODOs. I'm trying to implement some. This is my first small attempt. Any suggestion? Thank you! Regards, Cheng # HG changeset patch # User Han Cheng # Date 1396856176 -28800 # Mon Apr 07 15:36:16 2014 +0800 # Node ID de04aae10531d3bae43804231ed3566ececec481 # Parent 0c0dd1aacdf55f3422cfa2edd4dfe85f4d0d8b34 Core: configurable listening try number diff -r 0c0dd1aacdf5 -r de04aae10531 src/core/nginx.c --- a/src/core/nginx.c Tue Apr 01 20:53:18 2014 +0400 +++ b/src/core/nginx.c Mon Apr 07 15:36:16 2014 +0800 @@ -23,6 +23,8 @@ void *conf); static char *ngx_set_worker_processes(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); +static char *ngx_set_listening_tries(ngx_conf_t *cf, ngx_command_t *cmd, + void *conf); static ngx_conf_enum_t ngx_debug_points[] = { @@ -83,6 +85,13 @@ offsetof(ngx_core_conf_t, debug_points), &ngx_debug_points }, + { ngx_string("listening_tries"), + NGX_MAIN_CONF|NGX_DIRECT_CONF|NGX_CONF_TAKE1, + ngx_set_listening_tries, + 0, + 0, + NULL }, + { ngx_string("user"), NGX_MAIN_CONF|NGX_DIRECT_CONF|NGX_CONF_TAKE12, ngx_set_user, @@ -952,6 +961,8 @@ ccf->worker_processes = NGX_CONF_UNSET; ccf->debug_points = NGX_CONF_UNSET; + ccf->listening_tries = NGX_CONF_UNSET; + ccf->rlimit_nofile = NGX_CONF_UNSET; ccf->rlimit_core = NGX_CONF_UNSET; ccf->rlimit_sigpending = NGX_CONF_UNSET; @@ -986,6 +997,8 @@ ngx_conf_init_value(ccf->worker_processes, 1); ngx_conf_init_value(ccf->debug_points, 0); + ngx_conf_init_value(ccf->listening_tries, 5); + #if (NGX_HAVE_CPU_AFFINITY) if (ccf->cpu_affinity_n @@ -1361,3 +1374,29 @@ return NGX_CONF_OK; } + + +static char * +ngx_set_listening_tries(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) +{ + ngx_str_t *value; + ngx_core_conf_t *ccf; + + ccf = (ngx_core_conf_t *) conf; + + if (ccf->listening_tries != NGX_CONF_UNSET) { + return "is duplicate"; + } + + value = (ngx_str_t *) cf->args->elts; + + ccf->listening_tries = ngx_atoi(value[1].data, value[1].len); + + if (ccf->listening_tries == NGX_ERROR) { + return "invalid value"; + } else if (ccf->listening_tries > 1024) { + return "too much tries"; + } + + return NGX_CONF_OK; +} diff -r 0c0dd1aacdf5 -r de04aae10531 src/core/ngx_connection.c --- a/src/core/ngx_connection.c Tue Apr 01 20:53:18 2014 +0400 +++ b/src/core/ngx_connection.c Mon Apr 07 15:36:16 2014 +0800 @@ -305,11 +305,13 @@ ngx_open_listening_sockets(ngx_cycle_t *cycle) { int reuseaddr; - ngx_uint_t i, tries, failed; + ngx_uint_t i, failed; + ngx_int_t tries; ngx_err_t err; ngx_log_t *log; ngx_socket_t s; ngx_listening_t *ls; + ngx_core_conf_t *ccf; reuseaddr = 1; #if (NGX_SUPPRESS_WARN) @@ -318,9 +320,9 @@ log = cycle->log; - /* TODO: configurable try number */ + ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_core_module); - for (tries = 5; tries; tries--) { + for (tries = ccf->listening_tries; tries; tries--) { failed = 0; /* for each listening socket */ diff -r 0c0dd1aacdf5 -r de04aae10531 src/core/ngx_cycle.h --- a/src/core/ngx_cycle.h Tue Apr 01 20:53:18 2014 +0400 +++ b/src/core/ngx_cycle.h Mon Apr 07 15:36:16 2014 +0800 @@ -81,6 +81,8 @@ ngx_int_t worker_processes; ngx_int_t debug_points; + ngx_int_t listening_tries; + ngx_int_t rlimit_nofile; ngx_int_t rlimit_sigpending; off_t rlimit_core; From fdasilvayy at gmail.com Mon Apr 7 08:35:33 2014 From: fdasilvayy at gmail.com (Filipe Da Silva) Date: Mon, 7 Apr 2014 10:35:33 +0200 Subject: [PATCH] Mail: added support for SSL client certificate In-Reply-To: References: <5c7ccfc96070fc8b5d77.1392983328@FLEVIONNOIS2.dictao.com> <20140307113147.GY34696@mdounin.ru> Message-ID: Hi, >From the mail-auth-http module point of view, the Auth-Verify is a trivial information. Its value mostly depends of the current server configuration ( verify setting ). IMHO, it could be discard. About the various/duplicated headers related to the client certificate, a smart solution could be adding a 'auth_http_client_cert' setting. It could be either a kind of bit-field allowing to select the wanted headers one by one or a log level. Bit-field doesn't seems to be a part of nginx configuration usages. Instead, a short list of keywords could be defined, may be following the OpenSSL display one: http://www.openssl.org/docs/apps/x509.html#DISPLAY_OPTIONS Or, the auth_http_client_cert log levels could be : - none - basic -> just the Certificate Subject - detailed : Subject, Issuer - complete : Subject, Issuer, sha1 hash - full -> whole certificate IMHO, 'detailled' should be the default settings, if not configured. Regards, Filipe da Silva 2014-03-18 18:40 GMT+01:00 Franck Levionnois : > Hello, > > It doesn't seem to exist a standard for this header name. Apache and F5 let > the user choose it, but this make the configuration more complicated. I > don't think that the name is a problem, because it can be set on the > authorization server. > > If the certificate is transmited, all other informations are duplicated > (except Auth-Verify). Forwarding the certificate is the most usefull, > because it can be used to make controls on its properties. > > Kind regards, > Franck Levionnois. > > > > 2014-03-07 12:31 GMT+01:00 Maxim Dounin : > >> Hello! >> >> On Fri, Mar 07, 2014 at 09:40:11AM +0100, Franck Levionnois wrote: >> >> > Hello, >> > I haven't seen any comment on this patch. Is it ok for you ? >> >> Sorry, I haven't yet had a time to look into it in detail. >> >> Most problematic part is still auth_http protocol changes - in >> particular, headers send and names used for them. I tend to think >> there should be better names, and probably we can safely omit some >> information as duplicate/unneeded. >> >> -- >> Maxim Dounin >> http://nginx.org/ >> >> _______________________________________________ >> nginx-devel mailing list >> nginx-devel at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > > From mdounin at mdounin.ru Mon Apr 7 11:27:47 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 7 Apr 2014 15:27:47 +0400 Subject: [PATCH] SSL: support automatic selection of ECDH temporary key parameters In-Reply-To: References: <20140401170037.GZ34696@mdounin.ru> Message-ID: <20140407112747.GZ34696@mdounin.ru> Hello! On Sun, Apr 06, 2014 at 07:09:41PM -0700, Piotr Sikora wrote: > Hey Maxim, > > > Comments about the patch below, in no particular order: > > > > - Suggested code doesn't seem to allow to use the default list of > > curves, as normally available with just a call to > > SSL_CTX_set_ecdh_auto(); this seems to be what OpenSSL > > recommends to use by default, and we may want to follow. > > But isn't that the way nginx usually interacts with OpenSSL? i.e. > always calling SSL_CTX_set_cipher_list(), even with the default > "DEFAULT" value? The problem is that there is no equivalent "default" value in case of curves. But I think we can live with it, at least till OpenSSL folks will introduce one. > Also, right now nginx uses NIST P-256 curve. Calling > SSL_CTX_set_ecdh_auto() without limiting list of supported curves > enables all of them, with preference for the strongest ones, which > means that all modern browsers will start using NIST P-521 and clients > compiled against OpenSSL will start using NIST B-571. > > This results in rather significant performance loss: > > op op/s > 256 bit ecdh (nistp256) 0.0002s 4589.7 > 521 bit ecdh (nistp521) 0.0006s 1551.9 > 571 bit ecdh (nistb571) 0.0030s 330.9 > > and I don't think that it's a good idea to silently change used curve > for users using default configuration. Agree, preserving prime256v1 as the default looks reasonable. > > - Error messages in the ngx_ssl_ecdh_curve() are way off from > > what's normally used in ngx_event_openssl.c, and probably > > it's not a good idea to use similar messages in the new code. > > Right, my bad. > > > - If nginx was compiled with OpenSSL 1.0.2, but used with an > > older version, things will not work at all; this is not something > > completely unacceptable, but it's something we may want to > > avoid. > > Will look into it. > > > - SSL_CTX_set_options(SSL_OP_SINGLE_ECDH_USE) is not used > > with OpenSSL 1.0.2, and this looks just wrong. > > Well, it just looks wrong ;) > > If you dig into OpenSSL's code, you'll notice that this option is > checked only in 3 places: when setting SSL_CTX_set_tmp_ecdh() / > SSL_set_tmp_ecdh() to see if we need to generate "persistent" > temporary key and during ServerKeyExchange phase to see if we need to > generate ephemeral temporary key. This check consists of 3 conditions > and first two are fulfilled via SSL_CTX_set_ecdh_auto(), so > SSL_OP_SINGLE_ECDH_USE is never check when using it. While the SSL_OP_SINGLE_ECDH_USE may be unneeded with current OpenSSL code in case of SSL_CTX_set_ecdh_auto(), I would prefer to keep the option set anyway - as nothing stops OpenSSL from implementing a key cache for this case. -- Maxim Dounin http://nginx.org/ From mdounin at mdounin.ru Mon Apr 7 13:46:39 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 7 Apr 2014 17:46:39 +0400 Subject: [PATCH] Core: configurable listening try number In-Reply-To: References: Message-ID: <20140407134639.GH34696@mdounin.ru> Hello! On Mon, Apr 07, 2014 at 04:34:52PM +0800, Han Cheng wrote: > > Hello, there, > > I'm a new guy to nginx. > Recently, I'm reading the source code. I found some TODOs. I'm trying to > implement some. This is my first small attempt. > > Any suggestion? It doesn't looks like it actually needs to be configurable. I don't recall any single request to make this configurable, nor any single use case where it may help. -- Maxim Dounin http://nginx.org/ From mdounin at mdounin.ru Mon Apr 7 15:18:25 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 07 Apr 2014 15:18:25 +0000 Subject: [nginx] Win32: fixed link flags with MSVC, broken by bfe536716dbf. Message-ID: details: http://hg.nginx.org/nginx/rev/ce98b460606e branches: changeset: 5639:ce98b460606e user: Maxim Dounin date: Mon Apr 07 18:29:05 2014 +0400 description: Win32: fixed link flags with MSVC, broken by bfe536716dbf. Notably, "-debug" was omitted, resulting in an executable without debug symbols. diffstat: auto/cc/msvc | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff --git a/auto/cc/msvc b/auto/cc/msvc --- a/auto/cc/msvc +++ b/auto/cc/msvc @@ -106,7 +106,7 @@ fi # precompiled headers CORE_DEPS="$CORE_DEPS $NGX_OBJS/ngx_config.pch" -CORE_LINK="$NGX_OBJS/ngx_pch.obj" +CORE_LINK="$CORE_LINK $NGX_OBJS/ngx_pch.obj" NGX_PCH="$NGX_OBJS/ngx_config.pch" NGX_BUILD_PCH="-Ycngx_config.h -Fp$NGX_OBJS/ngx_config.pch" NGX_USE_PCH="-Yungx_config.h -Fp$NGX_OBJS/ngx_config.pch" From mdounin at mdounin.ru Mon Apr 7 15:18:27 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 07 Apr 2014 15:18:27 +0000 Subject: [nginx] Win32: fixed shared ssl_session_cache (ticket #528). Message-ID: details: http://hg.nginx.org/nginx/rev/4c6ceca4f5f7 branches: changeset: 5640:4c6ceca4f5f7 user: Maxim Dounin date: Mon Apr 07 18:55:57 2014 +0400 description: Win32: fixed shared ssl_session_cache (ticket #528). In a worker process shm_zone->data was set to NULL instead of a proper value extracted from shared memory. diffstat: src/event/ngx_event_openssl.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diffs (20 lines): diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c --- a/src/event/ngx_event_openssl.c +++ b/src/event/ngx_event_openssl.c @@ -1804,13 +1804,13 @@ ngx_ssl_session_cache_init(ngx_shm_zone_ return NGX_OK; } + shpool = (ngx_slab_pool_t *) shm_zone->shm.addr; + if (shm_zone->shm.exists) { - shm_zone->data = data; + shm_zone->data = shpool->data; return NGX_OK; } - shpool = (ngx_slab_pool_t *) shm_zone->shm.addr; - cache = ngx_slab_alloc(shpool, sizeof(ngx_ssl_session_cache_t)); if (cache == NULL) { return NGX_ERROR; From vbart at nginx.com Mon Apr 7 15:29:26 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Mon, 07 Apr 2014 15:29:26 +0000 Subject: [nginx] SPDY: better detect premature closing of stream. Message-ID: details: http://hg.nginx.org/nginx/rev/0aeb6f63d242 branches: changeset: 5641:0aeb6f63d242 user: Valentin Bartenev date: Mon Apr 07 19:27:56 2014 +0400 description: SPDY: better detect premature closing of stream. Following a24f88eff684, now the case when the FIN flag is set in SYN_STREAM is also covered. diffstat: src/http/ngx_http_spdy.c | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-) diffs (20 lines): diff -r 4c6ceca4f5f7 -r 0aeb6f63d242 src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Mon Apr 07 18:55:57 2014 +0400 +++ b/src/http/ngx_http_spdy.c Mon Apr 07 19:27:56 2014 +0400 @@ -2962,6 +2962,16 @@ ngx_http_spdy_run_request(ngx_http_reque return; } + if (r->headers_in.content_length_n > 0 && r->spdy_stream->in_closed) { + ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, + "client prematurely closed stream"); + + r->spdy_stream->skip_data = NGX_SPDY_DATA_ERROR; + + ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST); + return; + } + ngx_http_process_request(r); } From vbart at nginx.com Mon Apr 7 15:29:27 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Mon, 07 Apr 2014 15:29:27 +0000 Subject: [nginx] SPDY: refactored ngx_http_spdy_state_read_data(). Message-ID: details: http://hg.nginx.org/nginx/rev/d2ac5cf4056d branches: changeset: 5642:d2ac5cf4056d user: Valentin Bartenev date: Mon Apr 07 19:27:56 2014 +0400 description: SPDY: refactored ngx_http_spdy_state_read_data(). There's no more need in a separate indicator of frame completeness after d74889fbf06d. diffstat: src/http/ngx_http_spdy.c | 9 ++------- 1 files changed, 2 insertions(+), 7 deletions(-) diffs (34 lines): diff -r 0aeb6f63d242 -r d2ac5cf4056d src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Mon Apr 07 19:27:56 2014 +0400 +++ b/src/http/ngx_http_spdy.c Mon Apr 07 19:27:56 2014 +0400 @@ -1498,7 +1498,6 @@ ngx_http_spdy_state_read_data(ngx_http_s ssize_t n; ngx_buf_t *buf; ngx_int_t rc; - ngx_uint_t complete; ngx_temp_file_t *tf; ngx_http_request_t *r; ngx_http_spdy_stream_t *stream; @@ -1523,12 +1522,8 @@ ngx_http_spdy_state_read_data(ngx_http_s size = end - pos; - if (size >= sc->length) { + if (size > sc->length) { size = sc->length; - complete = 1; - - } else { - complete = 0; } r = stream->request; @@ -1600,7 +1595,7 @@ ngx_http_spdy_state_read_data(ngx_http_s r->request_length += size; } - if (!complete) { + if (sc->length) { return ngx_http_spdy_state_save(sc, pos, end, ngx_http_spdy_state_read_data); } From vbart at nginx.com Mon Apr 7 15:29:29 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Mon, 07 Apr 2014 15:29:29 +0000 Subject: [nginx] SPDY: consistently handle control frames with unknown type. Message-ID: details: http://hg.nginx.org/nginx/rev/436f3605195a branches: changeset: 5643:436f3605195a user: Valentin Bartenev date: Mon Apr 07 19:27:56 2014 +0400 description: SPDY: consistently handle control frames with unknown type. The SPDY draft 2 specification requires that if an endpoint receives a control frame for a type it does not recognize, it must ignore the frame. But the 3 and 3.1 drafts don't seem to declare any behavior for such case. Then sticking with the previous draft in this matter looks to be right. But previously, only 8 least significant bits of the type field were parsed while the rest of 16 bits of the field were checked against zero. Though there are no known frame types bigger than 255, this resulted in inconsistency in handling of such frames: they were not recognized as valid frames at all, and the connection was closed. diffstat: src/http/ngx_http_spdy.c | 15 ++++++++++----- 1 files changed, 10 insertions(+), 5 deletions(-) diffs (49 lines): diff -r d2ac5cf4056d -r 436f3605195a src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Mon Apr 07 19:27:56 2014 +0400 +++ b/src/http/ngx_http_spdy.c Mon Apr 07 19:27:56 2014 +0400 @@ -47,11 +47,11 @@ #define ngx_spdy_ctl_frame_check(h) \ - (((h) & 0xffffff00) == ngx_spdy_ctl_frame_head(0)) + (((h) & 0xffff0000) == ngx_spdy_ctl_frame_head(0)) #define ngx_spdy_data_frame_check(h) \ (!((h) & (uint32_t) NGX_SPDY_CTL_BIT << 31)) -#define ngx_spdy_ctl_frame_type(h) ((h) & 0x000000ff) +#define ngx_spdy_ctl_frame_type(h) ((h) & 0x0000ffff) #define ngx_spdy_frame_flags(p) ((p) >> 24) #define ngx_spdy_frame_length(p) ((p) & 0x00ffffff) #define ngx_spdy_frame_id(p) ((p) & 0x00ffffff) @@ -836,7 +836,8 @@ static u_char * ngx_http_spdy_state_head(ngx_http_spdy_connection_t *sc, u_char *pos, u_char *end) { - uint32_t head, flen; + uint32_t head, flen; + ngx_uint_t type; if (end - pos < NGX_SPDY_FRAME_HEADER_SIZE) { return ngx_http_spdy_state_save(sc, pos, end, @@ -859,7 +860,9 @@ ngx_http_spdy_state_head(ngx_http_spdy_c head, sc->flags, sc->length); if (ngx_spdy_ctl_frame_check(head)) { - switch (ngx_spdy_ctl_frame_type(head)) { + type = ngx_spdy_ctl_frame_type(head); + + switch (type) { case NGX_SPDY_SYN_STREAM: return ngx_http_spdy_state_syn_stream(sc, pos, end); @@ -885,7 +888,9 @@ ngx_http_spdy_state_head(ngx_http_spdy_c case NGX_SPDY_WINDOW_UPDATE: return ngx_http_spdy_state_window_update(sc, pos, end); - default: /* TODO logging */ + default: + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, + "spdy control frame with unknown type %ui", type); return ngx_http_spdy_state_skip(sc, pos, end); } } From hc0606 at gmail.com Mon Apr 7 15:44:18 2014 From: hc0606 at gmail.com (Han Cheng) Date: Mon, 7 Apr 2014 23:44:18 +0800 Subject: [PATCH] Core: configurable listening try number In-Reply-To: <20140407134639.GH34696@mdounin.ru> References: <20140407134639.GH34696@mdounin.ru> Message-ID: Hello, Maxim, As I'm new to nginx, please point out if I'm wrong. The reason why we should have multiply tries is we may fail when we bind(). That is because the address and port are in use(temporary which caused by unclean close or others, permanent like another nginx is using) or others. In the temporary use case, we may wait and retry. Configurable retry make us more probable succeed in that case. By the way, one other TODO in the source code is the configurable wait time. I think we do not need it when we have retry. Anyway, in most cases, we succeed in the first try and go on. Thanks 2014-04-07 21:46 GMT+08:00 Maxim Dounin : > Hello! > > On Mon, Apr 07, 2014 at 04:34:52PM +0800, Han Cheng wrote: > > > > > Hello, there, > > > > I'm a new guy to nginx. > > Recently, I'm reading the source code. I found some TODOs. I'm trying to > > implement some. This is my first small attempt. > > > > Any suggestion? > > It doesn't looks like it actually needs to be configurable. I > don't recall any single request to make this configurable, nor any > single use case where it may help. > > -- > Maxim Dounin > http://nginx.org/ > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -- Best regards, Cheng -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Mon Apr 7 17:06:32 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 7 Apr 2014 21:06:32 +0400 Subject: [PATCH] Core: configurable listening try number In-Reply-To: References: <20140407134639.GH34696@mdounin.ru> Message-ID: <20140407170632.GQ34696@mdounin.ru> Hello! On Mon, Apr 07, 2014 at 11:44:18PM +0800, Han Cheng wrote: > Hello, Maxim, > > As I'm new to nginx, please point out if I'm wrong. > > The reason why we should have multiply tries is we may fail when we bind(). > That is because the address and port > are in use(temporary which caused by unclean close or others, permanent > like another nginx is using) or others. > > In the temporary use case, we may wait and retry. Configurable retry make > us more probable succeed in that case. While in theory this may be usefull when, e.g., switching from other servers to nginx, it's not something expected to happen during normal operation. Because of this there is no real need to configure retries - if nginx fails to start because previously working server isn't stopped in time, administrator will be able to start nginx again. And if this happens for some reason during normal operation, probably the problem is anyway fatal and retries won't help. -- Maxim Dounin http://nginx.org/ From vbart at nginx.com Tue Apr 8 09:19:48 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Tue, 08 Apr 2014 09:19:48 +0000 Subject: [nginx] SPDY: avoid creating flush frames. Message-ID: details: http://hg.nginx.org/nginx/rev/b948f9c73111 branches: changeset: 5644:b948f9c73111 user: Valentin Bartenev date: Mon Apr 07 23:35:33 2014 +0400 description: SPDY: avoid creating flush frames. Previously, an empty frame object was created for an output chain that contains only sync or flush empty buffers. But since 39d7eef2e332 every DATA frame has the flush flag set on its last buffer, so there's no need any more in additional flush buffers in the output queue and they can be skipped. Note that such flush frames caused an incorrect $body_bytes_sent value. diffstat: src/http/ngx_http_spdy_filter_module.c | 79 ++++++++++++++++++--------------- 1 files changed, 44 insertions(+), 35 deletions(-) diffs (112 lines): diff -r 436f3605195a -r b948f9c73111 src/http/ngx_http_spdy_filter_module.c --- a/src/http/ngx_http_spdy_filter_module.c Mon Apr 07 19:27:56 2014 +0400 +++ b/src/http/ngx_http_spdy_filter_module.c Mon Apr 07 23:35:33 2014 +0400 @@ -625,6 +625,20 @@ ngx_http_spdy_send_chain(ngx_connection_ r = fc->data; stream = r->spdy_stream; +#if (NGX_SUPPRESS_WARN) + size = 0; +#endif + + while (in) { + size = ngx_buf_size(in->buf); + + if (size || in->buf->last_buf) { + break; + } + + in = in->next; + } + if (in == NULL) { if (stream->queued) { @@ -638,8 +652,6 @@ ngx_http_spdy_send_chain(ngx_connection_ sc = stream->connection; - size = ngx_buf_size(in->buf); - if (size && ngx_http_spdy_flow_control(sc, stream) == NGX_DECLINED) { fc->write->delayed = 1; return in; @@ -850,48 +862,45 @@ ngx_http_spdy_filter_get_data_frame(ngx_ "spdy:%ui create DATA frame %p: len:%uz flags:%ui", stream->id, frame, len, flags); - if (len || flags) { + cl = ngx_chain_get_free_buf(stream->request->pool, + &stream->free_data_headers); + if (cl == NULL) { + return NULL; + } - cl = ngx_chain_get_free_buf(stream->request->pool, - &stream->free_data_headers); - if (cl == NULL) { + buf = cl->buf; + + if (buf->start) { + p = buf->start; + buf->pos = p; + + p += NGX_SPDY_SID_SIZE; + + (void) ngx_spdy_frame_write_flags_and_len(p, flags, len); + + } else { + p = ngx_palloc(stream->request->pool, NGX_SPDY_FRAME_HEADER_SIZE); + if (p == NULL) { return NULL; } - buf = cl->buf; + buf->pos = p; + buf->start = p; - if (buf->start) { - p = buf->start; - buf->pos = p; + p = ngx_spdy_frame_write_sid(p, stream->id); + p = ngx_spdy_frame_write_flags_and_len(p, flags, len); - p += NGX_SPDY_SID_SIZE; + buf->last = p; + buf->end = p; - (void) ngx_spdy_frame_write_flags_and_len(p, flags, len); + buf->tag = (ngx_buf_tag_t) &ngx_http_spdy_filter_get_data_frame; + buf->memory = 1; + } - } else { - p = ngx_palloc(stream->request->pool, NGX_SPDY_FRAME_HEADER_SIZE); - if (p == NULL) { - return NULL; - } + cl->next = first; + first = cl; - buf->pos = p; - buf->start = p; - - p = ngx_spdy_frame_write_sid(p, stream->id); - p = ngx_spdy_frame_write_flags_and_len(p, flags, len); - - buf->last = p; - buf->end = p; - - buf->tag = (ngx_buf_tag_t) &ngx_http_spdy_filter_get_data_frame; - buf->memory = 1; - } - - cl->next = first; - first = cl; - - last->buf->flush = 1; - } + last->buf->flush = 1; frame->first = first; frame->last = last; From mdounin at mdounin.ru Tue Apr 8 14:05:42 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 08 Apr 2014 14:05:42 +0000 Subject: [nginx] Updated OpenSSL used for win32 builds. Message-ID: details: http://hg.nginx.org/nginx/rev/7a7ced5b462c branches: changeset: 5645:7a7ced5b462c user: Maxim Dounin date: Tue Apr 08 17:48:03 2014 +0400 description: Updated OpenSSL used for win32 builds. diffstat: misc/GNUmakefile | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff --git a/misc/GNUmakefile b/misc/GNUmakefile --- a/misc/GNUmakefile +++ b/misc/GNUmakefile @@ -5,7 +5,7 @@ NGINX = nginx-$(VER) TEMP = tmp OBJS = objs.msvc8 -OPENSSL = openssl-1.0.1f +OPENSSL = openssl-1.0.1g ZLIB = zlib-1.2.8 PCRE = pcre-8.34 From mdounin at mdounin.ru Tue Apr 8 14:21:55 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 08 Apr 2014 14:21:55 +0000 Subject: [nginx] nginx-1.5.13-RELEASE Message-ID: details: http://hg.nginx.org/nginx/rev/fd722b890eab branches: changeset: 5646:fd722b890eab user: Maxim Dounin date: Tue Apr 08 18:15:21 2014 +0400 description: nginx-1.5.13-RELEASE diffstat: docs/xml/nginx/changes.xml | 93 ++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 93 insertions(+), 0 deletions(-) diffs (103 lines): diff --git a/docs/xml/nginx/changes.xml b/docs/xml/nginx/changes.xml --- a/docs/xml/nginx/changes.xml +++ b/docs/xml/nginx/changes.xml @@ -5,6 +5,99 @@ + + + + +???????? ????????? ???-??????; +? ?????????? variables_hash_max_size ? types_hash_bucket_size +???????? ?? ????????? ???????? ?? 1024 ? 64 ??????????????. + + +improved hash table handling; +the default values of the "variables_hash_max_size" and +"types_hash_bucket_size" were changed to 1024 and 64 respectively. + + + + + +?????? ngx_http_mp4_module ?????? ???????? ???????? end. + + +the ngx_http_mp4_module now supports the "end" argument. + + + + + +????????? byte ranges ??????? ngx_http_mp4_module ? ??? ?????????? +??????? ? ???. + + +byte ranges support in the ngx_http_mp4_module and while saving responses +to cache. + + + + + +?????? nginx ?? ????? ? ??? ????????? "ngx_slab_alloc() failed: no memory" +??? ????????????? ??????????? ?????? ? ssl_session_cache +? ? ?????? ngx_http_limit_req_module. + + +alerts "ngx_slab_alloc() failed: no memory" no longer logged +when using shared memory in the "ssl_session_cache" directive +and in the ngx_http_limit_req_module. + + + + + +????????? underscores_in_headers +?? ????????? ????????????? ? ?????? ??????? ?????????.
+??????? Piotr Sikora. +
+ +the "underscores_in_headers" directive +did not allow underscore as a first character of a header.
+Thanks to Piotr Sikora. +
+
+ + + +cache manager ??? ????????? ????????? ??? ?????? ? nginx/Windows. + + +cache manager might hog CPU on exit in nginx/Windows. + + + + + +??? ????????????? ssl_session_cache ? ?????????? shared +??????? ??????? nginx/Windows ?????????? ????????. + + +nginx/Windows terminated abnormally +if the "ssl_session_cache" directive was used with the "shared" parameter. + + + + + +? ?????? ngx_http_spdy_module. + + +in the ngx_http_spdy_module. + + + +
+ + From mdounin at mdounin.ru Tue Apr 8 14:21:56 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 08 Apr 2014 14:21:56 +0000 Subject: [nginx] release-1.5.13 tag Message-ID: details: http://hg.nginx.org/nginx/rev/fe35c373ef73 branches: changeset: 5647:fe35c373ef73 user: Maxim Dounin date: Tue Apr 08 18:15:22 2014 +0400 description: release-1.5.13 tag diffstat: .hgtags | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (8 lines): diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -368,3 +368,4 @@ 5a1759f33b7fa6270e1617c08d7e655b7b127f26 b798fc020e3a84ef68e6c9f47865a319c826d33c release-1.5.10 f995a10d4c7e9a817157a6ce7b753297ad32897e release-1.5.11 97b47d95e4449cbde976657cf8cbbc118351ffe0 release-1.5.12 +fd722b890eabc600394349730a093f50dac31639 release-1.5.13 From vbart at nginx.com Tue Apr 8 16:16:19 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Tue, 08 Apr 2014 16:16:19 +0000 Subject: [nginx] Version bump. Message-ID: details: http://hg.nginx.org/nginx/rev/10915ad539b4 branches: changeset: 5648:10915ad539b4 user: Valentin Bartenev date: Tue Apr 08 20:11:31 2014 +0400 description: Version bump. diffstat: src/core/nginx.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (14 lines): diff -r fe35c373ef73 -r 10915ad539b4 src/core/nginx.h --- a/src/core/nginx.h Tue Apr 08 18:15:22 2014 +0400 +++ b/src/core/nginx.h Tue Apr 08 20:11:31 2014 +0400 @@ -9,8 +9,8 @@ #define _NGINX_H_INCLUDED_ -#define nginx_version 1005013 -#define NGINX_VERSION "1.5.13" +#define nginx_version 1005014 +#define NGINX_VERSION "1.5.14" #define NGINX_VER "nginx/" NGINX_VERSION #define NGINX_VAR "NGINX" From vbart at nginx.com Tue Apr 8 16:16:20 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Tue, 08 Apr 2014 16:16:20 +0000 Subject: [nginx] SPDY: fixed arguments supplied for an error message. Message-ID: details: http://hg.nginx.org/nginx/rev/e1dcb983d6b3 branches: changeset: 5649:e1dcb983d6b3 user: Valentin Bartenev date: Tue Apr 08 20:12:30 2014 +0400 description: SPDY: fixed arguments supplied for an error message. diffstat: src/http/ngx_http_spdy.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r 10915ad539b4 -r e1dcb983d6b3 src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Tue Apr 08 20:11:31 2014 +0400 +++ b/src/http/ngx_http_spdy.c Tue Apr 08 20:12:30 2014 +0400 @@ -1421,7 +1421,7 @@ ngx_http_spdy_state_data(ngx_http_spdy_c ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, "client violated connection flow control: length of " "received DATA frame %uz, while available window %uz", - stream->id, sc->length, sc->recv_window); + sc->length, sc->recv_window); return ngx_http_spdy_state_protocol_error(sc); } From majek04 at gmail.com Wed Apr 9 15:53:04 2014 From: majek04 at gmail.com (Marek Majkowski) Date: Wed, 9 Apr 2014 16:53:04 +0100 Subject: [patch] Set SO_REUSEADDR on outgoing TCP connections Message-ID: Usually, when establishing a connection the kernel allocates outgoing TCP/IP port automatically from an ephemeral port range. Unfortunately when selecting the outgoing source IP (using bind before connect) the kernel needs a unique port number. As the result it can only establish a single outgoing connection from a single source port. This can cause problems with a large number of outgoing proxy connections - it's possible for the kernel to run out free ports in the ephemeral range. The situation can be improved - TCP/IP allows any number of connections to share outgoing TCP/IP port and host pair assuming the destination addresses differ. This patch sets a SO_REUSEADDR flag on the connections that use bind before connect to select ougoing source address. This will allow the kernel to reuse source port numbers, given that the destination addresses are different. The patch will work perfectly well assuming there aren't too many connections to one destination address and port. If that happens the kernel may randomly allocate an outgoing port number that is already used for a given destination and attempt to connect() will fail with EADDRNOTAVAIL. This is fairly easy to detect, and we can just retry connecting again, using another random source port allocated by the kernel. Unfortunately it introduces some nondeterminism, in an extreme situation a connection attempt may fail while we still have a theoretical chance of success. This situation is not worse than what we have right now: currently the number of outgoing ports is strongly limited by a size of ephemeral port range. With this patch it's possible to establish pretty much unlimited number of outgoing connections, assuming there are many destinations. To work around the situation of thousands connections to the same destination address, we will retry connection a few times before giving up. The patch hardcodes a retry count of 8, which I believe strikes the right balance between the probability of success and the cost of retrying socket allocation. Assuming 1 connection already present to exactly the same destination, the probability of collision is 1/ephemeral_port_range given no retry attempts. Given 8 retries we get following numbers: * If 1% of ephemeral_ports are busy with given destination address, eight retry attempts will fail for a one connection in 9999999999999998. * For 10%: one in 100000000 * For 50%: one in 256 Finally, during the last retry run we do *not* set the SO_REUSEADDR flag, making sure the kernel really doesn't have any free port left. Unfortunately there is a side effect to not setting this flag: we limit the outgoing port range for further connections, as source ports without SO_REUSEADDR can't be reused. Copy of the patch: https://gist.github.com/anonymous/10285483 -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Set-SO_REUSEADDR-on-outgoing-TCP-connections.patch Type: application/octet-stream Size: 5826 bytes Desc: not available URL: From majek04 at gmail.com Wed Apr 9 16:02:11 2014 From: majek04 at gmail.com (Marek Majkowski) Date: Wed, 9 Apr 2014 17:02:11 +0100 Subject: [patch] Set SO_REUSEADDR on outgoing TCP connections In-Reply-To: References: Message-ID: Testing the patch is straightforward once you have an idea how to do it. 1) Reduce ephemeral port range $ sudo -s bash -c "echo 32000 32004 > /proc/sys/net/ipv4/ip_local_port_range" 2) Start nginx with simple config from this gist. https://gist.github.com/anonymous/10285909 3) The gist also contains two simple python scripts. They establish connections to local nginx instance, and _don't_ use ephemeral port range, therefore not interfere with the mechanisms described in this patch. Run the scripts against patched and unpatched nginx. The first one connects to multiple destinations: $ python connect_multi.py You should see that patched nginx can indeed establish more connection before running out of ports. $ python connect_single.py You should see that both patched and unpatched nginx are able to open exactly the same number of connections to a single destination. 4) Looking at outgoing SYN's and FIN's will help debugging: $ sudo tcpdump -ni any 'port 80 and ((tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0) or (tcp[tcpflags] & tcp-fin != 0))' 5) Running nginx with strace could also give some hints strace -f -e trace=bind,connect,close,setsockopt,socket ./objs/nginx .... Marek On Wed, Apr 9, 2014 at 4:53 PM, Marek Majkowski wrote: > Usually, when establishing a connection the kernel allocates outgoing > TCP/IP port automatically from an ephemeral port range. Unfortunately > when selecting the outgoing source IP (using bind before connect) the > kernel needs a unique port number. As the result it can only establish > a single outgoing connection from a single source port. This can cause > problems with a large number of outgoing proxy connections - it's > possible for the kernel to run out free ports in the ephemeral range. From vbart at nginx.com Thu Apr 10 13:49:50 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Thu, 10 Apr 2014 13:49:50 +0000 Subject: [nginx] SPDY: moved a variable initialization near to its check. Message-ID: details: http://hg.nginx.org/nginx/rev/363395795384 branches: changeset: 5650:363395795384 user: Valentin Bartenev date: Wed Apr 09 18:15:32 2014 +0400 description: SPDY: moved a variable initialization near to its check. This should prevent attempts of using pointer before it was checked, since all modern compilers are able to spot access to uninitialized variable. No functional changes. diffstat: src/http/ngx_http_spdy.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (21 lines): diff -r e1dcb983d6b3 -r 363395795384 src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Tue Apr 08 20:12:30 2014 +0400 +++ b/src/http/ngx_http_spdy.c Wed Apr 09 18:15:32 2014 +0400 @@ -1412,8 +1412,6 @@ ngx_http_spdy_state_data(ngx_http_spdy_c { ngx_http_spdy_stream_t *stream; - stream = sc->stream; - ngx_log_debug0(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, "spdy DATA frame"); @@ -1441,6 +1439,8 @@ ngx_http_spdy_state_data(ngx_http_spdy_c sc->recv_window = NGX_SPDY_MAX_WINDOW; } + stream = sc->stream; + if (stream == NULL) { return ngx_http_spdy_state_skip(sc, pos, end); } From mdounin at mdounin.ru Thu Apr 10 15:40:14 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 10 Apr 2014 19:40:14 +0400 Subject: [patch] Set SO_REUSEADDR on outgoing TCP connections In-Reply-To: References: Message-ID: <20140410154014.GA34696@mdounin.ru> Hello! On Wed, Apr 09, 2014 at 04:53:04PM +0100, Marek Majkowski wrote: > Usually, when establishing a connection the kernel allocates outgoing > TCP/IP port automatically from an ephemeral port range. Unfortunately > when selecting the outgoing source IP (using bind before connect) the > kernel needs a unique port number. As the result it can only establish > a single outgoing connection from a single source port. This can cause > problems with a large number of outgoing proxy connections - it's > possible for the kernel to run out free ports in the ephemeral range. > > The situation can be improved - TCP/IP allows any number of > connections to share outgoing TCP/IP port and host pair assuming the > destination addresses differ. > > This patch sets a SO_REUSEADDR flag on the connections that use bind > before connect to select ougoing source address. This will allow the > kernel to reuse source port numbers, given that the destination > addresses are different. > > The patch will work perfectly well assuming there aren't too many > connections to one destination address and port. If that happens the > kernel may randomly allocate an outgoing port number that is already > used for a given destination and attempt to connect() will fail with > EADDRNOTAVAIL. This is fairly easy to detect, and we can just retry > connecting again, using another random source port allocated by the > kernel. While it may be interesting approach to overcome the limitation, I don't think that this is something that should be done by clients in real life. I think it's something kernel should care about, not clients. >From practical point of view, trivial solutions are to avoid bind() or use multiple addresses for bind(). -- Maxim Dounin http://nginx.org/ From majek04 at gmail.com Thu Apr 10 16:04:30 2014 From: majek04 at gmail.com (Marek Majkowski) Date: Thu, 10 Apr 2014 17:04:30 +0100 Subject: [patch] Set SO_REUSEADDR on outgoing TCP connections In-Reply-To: <20140410154014.GA34696@mdounin.ru> References: <20140410154014.GA34696@mdounin.ru> Message-ID: On Thu, Apr 10, 2014 at 4:40 PM, Maxim Dounin wrote: >> ... >> The patch will work perfectly well assuming there aren't too many >> connections to one destination address and port. If that happens the >> kernel may randomly allocate an outgoing port number that is already >> used for a given destination and attempt to connect() will fail with >> EADDRNOTAVAIL. This is fairly easy to detect, and we can just retry >> connecting again, using another random source port allocated by the >> kernel. > > While it may be interesting approach to overcome the limitation, I > don't think that this is something that should be done by clients > in real life. I think it's something kernel should care about, > not clients. > > From practical point of view, trivial solutions are to avoid > bind() or use multiple addresses for bind(). May I ask how can you specify multiple outgoing IP addresses for a single proxy? Indeed, this patch does add complexity to the ngx_event_connect_peer function. Unfortunately nginx currently supports bind before connect and in current form it severely reduces the number of possible outgoing connections. Using proxy_bind in current form is harmful. This patch fixes that. In perfect world kernel would be able to reuse ports established with bind before connect. Actually kernel can do that already - this is what the SO_REUSADDR flag is for. Unfortunately, due to BSD API compatibility kernel needs to allocate outgoing port on bind() - before it is told of the destination address. This inevitably may lead to conflicts indicated by EADDRNOTAVAIL error on connect(), and results in complexity in proper implementation of bind before connect. A reasonable solution to bind before connect limitations is to either merge this patch or retire proxy_bind option. Cheers, Marek From mdounin at mdounin.ru Thu Apr 10 17:50:13 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 10 Apr 2014 21:50:13 +0400 Subject: [patch] Set SO_REUSEADDR on outgoing TCP connections In-Reply-To: References: <20140410154014.GA34696@mdounin.ru> Message-ID: <20140410175013.GB34696@mdounin.ru> Hello! On Thu, Apr 10, 2014 at 05:04:30PM +0100, Marek Majkowski wrote: > On Thu, Apr 10, 2014 at 4:40 PM, Maxim Dounin wrote: > >> ... > >> The patch will work perfectly well assuming there aren't too many > >> connections to one destination address and port. If that happens the > >> kernel may randomly allocate an outgoing port number that is already > >> used for a given destination and attempt to connect() will fail with > >> EADDRNOTAVAIL. This is fairly easy to detect, and we can just retry > >> connecting again, using another random source port allocated by the > >> kernel. > > > > While it may be interesting approach to overcome the limitation, I > > don't think that this is something that should be done by clients > > in real life. I think it's something kernel should care about, > > not clients. > > > > From practical point of view, trivial solutions are to avoid > > bind() or use multiple addresses for bind(). > > May I ask how can you specify multiple outgoing IP addresses for a single proxy? There is more than one way to. Most trivial solution would be to use proxy_bind with a variable, and rotate the address in the variable somehow (split_clients, map, ...). > Indeed, this patch does add complexity to the ngx_event_connect_peer > function. Unfortunately nginx currently supports bind before connect > and in current form it severely reduces the number of possible > outgoing connections. Using proxy_bind in current form is harmful. > This patch fixes that. As suggest above, if it doesn't work for you - don't use it. (Actually, the bad thing is that on many OSes it's not possible to create more than 64k outgoing connections even without doing a bind(). And proxy_bind comes to rescue, despite the fact that it was originally implemented for a completely unrelated purpose.) > In perfect world kernel would be able to reuse ports established with > bind before connect. Actually kernel can do that already - this is > what the SO_REUSADDR flag is for. Unfortunately, due to BSD API > compatibility kernel needs to allocate outgoing port on bind() - > before it is told of the destination address. This inevitably may lead > to conflicts indicated by EADDRNOTAVAIL error on connect(), and > results in complexity in proper implementation of bind before connect. > > A reasonable solution to bind before connect limitations is to either > merge this patch or retire proxy_bind option. It's more or less obvious that kernel can't do proper source port selection in the "bind(), then connect()" scenario without some API extension (e.g., to allow to postpone source port allocation till destination address is known). Overcoming this limitation with SO_REUSEADDR and multiple retries looks wrong though. It's not a solution, it's at most workaround. And there are other workarounds available, see above. -- Maxim Dounin http://nginx.org/ From majek04 at gmail.com Thu Apr 10 22:20:29 2014 From: majek04 at gmail.com (Marek Majkowski) Date: Thu, 10 Apr 2014 23:20:29 +0100 Subject: [patch] Set SO_REUSEADDR on outgoing TCP connections In-Reply-To: <20140410175013.GB34696@mdounin.ru> References: <20140410154014.GA34696@mdounin.ru> <20140410175013.GB34696@mdounin.ru> Message-ID: On Thu, Apr 10, 2014 at 6:50 PM, Maxim Dounin wrote: > On Thu, Apr 10, 2014 at 05:04:30PM +0100, Marek Majkowski wrote: > >> On Thu, Apr 10, 2014 at 4:40 PM, Maxim Dounin wrote: >> >> ... >> >> The patch will work perfectly well assuming there aren't too many >> >> connections to one destination address and port. If that happens the >> >> kernel may randomly allocate an outgoing port number that is already >> >> used for a given destination and attempt to connect() will fail with >> >> EADDRNOTAVAIL. This is fairly easy to detect, and we can just retry >> >> connecting again, using another random source port allocated by the >> >> kernel. >> > >> > While it may be interesting approach to overcome the limitation, I >> > don't think that this is something that should be done by clients >> > in real life. I think it's something kernel should care about, >> > not clients. >> > >> > From practical point of view, trivial solutions are to avoid >> > bind() or use multiple addresses for bind(). >> >> May I ask how can you specify multiple outgoing IP addresses for a single proxy? > > There is more than one way to. Most trivial solution would be to > use proxy_bind with a variable, and rotate the address in the variable > somehow (split_clients, map, ...). > >> Indeed, this patch does add complexity to the ngx_event_connect_peer >> function. Unfortunately nginx currently supports bind before connect >> and in current form it severely reduces the number of possible >> outgoing connections. Using proxy_bind in current form is harmful. >> This patch fixes that. > > As suggest above, if it doesn't work for you - don't use it. > > (Actually, the bad thing is that on many OSes it's not possible to > create more than 64k outgoing connections even without doing a > bind(). And proxy_bind comes to rescue, despite the fact that it > was originally implemented for a completely unrelated purpose.) > >> In perfect world kernel would be able to reuse ports established with >> bind before connect. Actually kernel can do that already - this is >> what the SO_REUSADDR flag is for. Unfortunately, due to BSD API >> compatibility kernel needs to allocate outgoing port on bind() - >> before it is told of the destination address. This inevitably may lead >> to conflicts indicated by EADDRNOTAVAIL error on connect(), and >> results in complexity in proper implementation of bind before connect. >> >> A reasonable solution to bind before connect limitations is to either >> merge this patch or retire proxy_bind option. > > It's more or less obvious that kernel can't do proper source > port selection in the "bind(), then connect()" scenario without > some API extension (e.g., to allow to postpone source port allocation > till destination address is known). > > Overcoming this limitation with SO_REUSEADDR and multiple retries > looks wrong though. It's not a solution, it's at most workaround. > And there are other workarounds available, see above. How about yet another approach: maybe it's the nginx that should do the port allocation logic. We could transform the bind before connect thing to suggest a specific port to the kernel. This would reduce the complexity of code run in kernel as it wouldn't need to do allocation for us, just accept (or in rare cases reject) the bind attempt. Using SO_REUSEADDR flag will still be necessary to be able to reuse ports. The retry logic would need to be there, but the probability of retry will be extremaly low - as we are going to control the port allocation for nginx outgoing connections. We'd need to retry only if there is a conflict with a connection established by another process, which shouldn't happen too often unless the outgoing IP address is used by another application. Cheers, Marek From mdounin at mdounin.ru Fri Apr 11 12:26:58 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 11 Apr 2014 16:26:58 +0400 Subject: [patch] Set SO_REUSEADDR on outgoing TCP connections In-Reply-To: References: <20140410154014.GA34696@mdounin.ru> <20140410175013.GB34696@mdounin.ru> Message-ID: <20140411122658.GJ34696@mdounin.ru> Hello! On Thu, Apr 10, 2014 at 11:20:29PM +0100, Marek Majkowski wrote: > On Thu, Apr 10, 2014 at 6:50 PM, Maxim Dounin wrote: > > On Thu, Apr 10, 2014 at 05:04:30PM +0100, Marek Majkowski wrote: > > > >> On Thu, Apr 10, 2014 at 4:40 PM, Maxim Dounin wrote: > >> >> ... > >> >> The patch will work perfectly well assuming there aren't too many > >> >> connections to one destination address and port. If that happens the > >> >> kernel may randomly allocate an outgoing port number that is already > >> >> used for a given destination and attempt to connect() will fail with > >> >> EADDRNOTAVAIL. This is fairly easy to detect, and we can just retry > >> >> connecting again, using another random source port allocated by the > >> >> kernel. > >> > > >> > While it may be interesting approach to overcome the limitation, I > >> > don't think that this is something that should be done by clients > >> > in real life. I think it's something kernel should care about, > >> > not clients. > >> > > >> > From practical point of view, trivial solutions are to avoid > >> > bind() or use multiple addresses for bind(). > >> > >> May I ask how can you specify multiple outgoing IP addresses for a single proxy? > > > > There is more than one way to. Most trivial solution would be to > > use proxy_bind with a variable, and rotate the address in the variable > > somehow (split_clients, map, ...). > > > >> Indeed, this patch does add complexity to the ngx_event_connect_peer > >> function. Unfortunately nginx currently supports bind before connect > >> and in current form it severely reduces the number of possible > >> outgoing connections. Using proxy_bind in current form is harmful. > >> This patch fixes that. > > > > As suggest above, if it doesn't work for you - don't use it. > > > > (Actually, the bad thing is that on many OSes it's not possible to > > create more than 64k outgoing connections even without doing a > > bind(). And proxy_bind comes to rescue, despite the fact that it > > was originally implemented for a completely unrelated purpose.) > > > >> In perfect world kernel would be able to reuse ports established with > >> bind before connect. Actually kernel can do that already - this is > >> what the SO_REUSADDR flag is for. Unfortunately, due to BSD API > >> compatibility kernel needs to allocate outgoing port on bind() - > >> before it is told of the destination address. This inevitably may lead > >> to conflicts indicated by EADDRNOTAVAIL error on connect(), and > >> results in complexity in proper implementation of bind before connect. > >> > >> A reasonable solution to bind before connect limitations is to either > >> merge this patch or retire proxy_bind option. > > > > It's more or less obvious that kernel can't do proper source > > port selection in the "bind(), then connect()" scenario without > > some API extension (e.g., to allow to postpone source port allocation > > till destination address is known). > > > > Overcoming this limitation with SO_REUSEADDR and multiple retries > > looks wrong though. It's not a solution, it's at most workaround. > > And there are other workarounds available, see above. > > How about yet another approach: maybe it's the nginx that should do > the port allocation logic. We could transform the bind before connect > thing to suggest a specific port to the kernel. This would reduce the > complexity of code run in kernel as it wouldn't need to do allocation > for us, just accept (or in rare cases reject) the bind attempt. Using > SO_REUSEADDR flag will still be necessary to be able to reuse ports. > The retry logic would need to be there, but the probability of retry > will be extremaly low - as we are going to control the port allocation > for nginx outgoing connections. We'd need to retry only if there is a > conflict with a connection established by another process, which > shouldn't happen too often unless the outgoing IP address is used by > another application. This is just yet another workaround, and complex one. As suggested above, simpliest workaround is to don't use bind() at all. As long as you are using Linux, try "ip route add ... src
" instead; I've never tried, but it should be identical to "route ... -ifa
" on FreeBSD and shouldn't impose 64k source ports limit. Alternatively, if you want to solve the problem properly, it should be done in a kernel - likely by introducing something like SO_BINDLATER to allow bind() to postpone source port allocation till connect(). -- Maxim Dounin http://nginx.org/ From flevionnois at gmail.com Mon Apr 14 07:33:50 2014 From: flevionnois at gmail.com (Franck Levionnois) Date: Mon, 14 Apr 2014 09:33:50 +0200 Subject: [PATCH] Mail: added support for SSL client certificate In-Reply-To: References: <5c7ccfc96070fc8b5d77.1392983328@FLEVIONNOIS2.dictao.com> <20140307113147.GY34696@mdounin.ru> Message-ID: Hello, I forward Filipe's message, because it doesn't appear in forum's stack. I'm ok with the proposal. Kind Regards. Franck Levionnois. 2014-04-07 10:35 GMT+02:00 Filipe Da Silva : > Hi, > > From the mail-auth-http module point of view, the Auth-Verify is a > trivial information. > Its value mostly depends of the current server configuration ( verify > setting ). > IMHO, it could be discard. > > About the various/duplicated headers related to the client > certificate, a smart solution > could be adding a 'auth_http_client_cert' setting. > > It could be either a kind of bit-field allowing to select the wanted > headers one by one or a log level. > > Bit-field doesn't seems to be a part of nginx configuration usages. > Instead, a short list of keywords could be defined, may be following > the OpenSSL display one: > http://www.openssl.org/docs/apps/x509.html#DISPLAY_OPTIONS > > Or, the auth_http_client_cert log levels could be : > - none > - basic -> just the Certificate Subject > - detailed : Subject, Issuer > - complete : Subject, Issuer, sha1 hash > - full -> whole certificate > IMHO, 'detailled' should be the default settings, if not configured. > > Regards, > Filipe da Silva > > 2014-03-18 18:40 GMT+01:00 Franck Levionnois : > > Hello, > > > > It doesn't seem to exist a standard for this header name. Apache and F5 > let > > the user choose it, but this make the configuration more complicated. I > > don't think that the name is a problem, because it can be set on the > > authorization server. > > > > If the certificate is transmited, all other informations are duplicated > > (except Auth-Verify). Forwarding the certificate is the most usefull, > > because it can be used to make controls on its properties. > > > > Kind regards, > > Franck Levionnois. > > > > > > > > 2014-03-07 12:31 GMT+01:00 Maxim Dounin : > > > >> Hello! > >> > >> On Fri, Mar 07, 2014 at 09:40:11AM +0100, Franck Levionnois wrote: > >> > >> > Hello, > >> > I haven't seen any comment on this patch. Is it ok for you ? > >> > >> Sorry, I haven't yet had a time to look into it in detail. > >> > >> Most problematic part is still auth_http protocol changes - in > >> particular, headers send and names used for them. I tend to think > >> there should be better names, and probably we can safely omit some > >> information as duplicate/unneeded. > >> > >> -- > >> Maxim Dounin > >> http://nginx.org/ > >> > >> _______________________________________________ > >> nginx-devel mailing list > >> nginx-devel at nginx.org > >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From junwei.hjw at alibaba-inc.com Mon Apr 14 09:18:27 2014 From: junwei.hjw at alibaba-inc.com (=?GBK?B?uvq+/M6wKLLUw6Mp?=) Date: Mon, 14 Apr 2014 17:18:27 +0800 Subject: [PATCH] SPDY: added stream-ID checking for SYN_STREAM. Message-ID: <000301cf57c2$86325070$9296f150$@alibaba-inc.com> # SPDY patch # User Hu Junwei # Date 2014-04-14 11:27:39.304650193 +0800 # Node ID # Parent SPDY: added stream-ID checking for SVN_STREAM. The stream-ID should be checked whether it increases monotonically. diff -uNr a/src/http/ngx_http_spdy.c b/src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c 2014-04-12 10:58:44.444469431 +0800 +++ b/src/http/ngx_http_spdy.c 2014-04-14 11:27:39.304650193 +0800 @@ -940,10 +940,17 @@ sscf = ngx_http_get_module_srv_conf(sc->http_connection->conf_ctx, ngx_http_spdy_module); - if (sc->processing >= sscf->concurrent_streams) { - - ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, - "spdy concurrent streams excessed %ui", sc->processing); + if ((sid <= sc->last_sid) || (sc->processing >= sscf->concurrent_streams)) { + if (sid <= sc->last_sid) { + ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, + "spdy stream-ID does not increase monotonically. " + "(last=%ui, this=%ui)", + sc->last_sid, sid); + } else { + ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, + "spdy concurrent streams excessed %ui", + sc->processing); + } if (ngx_http_spdy_send_rst_stream(sc, sid, NGX_SPDY_REFUSED_STREAM, prio) From vbart at nginx.com Mon Apr 14 10:01:11 2014 From: vbart at nginx.com (Valentin V. Bartenev) Date: Mon, 14 Apr 2014 14:01:11 +0400 Subject: [PATCH] SPDY: added stream-ID checking for SYN_STREAM. In-Reply-To: <000301cf57c2$86325070$9296f150$@alibaba-inc.com> References: <000301cf57c2$86325070$9296f150$@alibaba-inc.com> Message-ID: <2837543.6GB55yqKMJ@vbart-laptop> On Monday 14 April 2014 17:18:27 ??? wrote: > # SPDY patch > # User Hu Junwei > # Date 2014-04-14 11:27:39.304650193 +0800 > # Node ID > # Parent > SPDY: added stream-ID checking for SVN_STREAM. > > The stream-ID should be checked whether it increases monotonically. > [..] While it would be good to have this check, but your patch doesn't follow specification (not to mention other problems in it). I already have a better one waiting in my patch queue: # HG changeset patch # User Valentin Bartenev # Date 1397147349 -14400 # Thu Apr 10 20:29:09 2014 +0400 # Node ID 2574215384c47374208fee3d36d5066f8f2d79b7 # Parent 3633957953840f8675511084d71eb28dcb3a1c5d SPDY: Stream-ID restrictions according to specification. diff -r 363395795384 -r 2574215384c4 src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Wed Apr 09 18:15:32 2014 +0400 +++ b/src/http/ngx_http_spdy.c Thu Apr 10 20:29:09 2014 +0400 @@ -937,6 +937,27 @@ ngx_http_spdy_state_syn_stream(ngx_http_ ngx_log_debug2(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, "spdy SYN_STREAM frame sid:%ui prio:%ui", sid, prio); + if (sid % 2 == 0 || sid <= sc->last_sid) { + ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, + "client sent SYN_STREAM frame " + "with invalid Stream-ID %ui", sid); + + stream = ngx_http_spdy_get_stream_by_id(sc, sid); + + if (stream) { + if (ngx_http_spdy_terminate_stream(sc, stream, + NGX_SPDY_PROTOCOL_ERROR) + != NGX_OK) + { + return ngx_http_spdy_state_internal_error(sc); + } + } + + return ngx_http_spdy_state_protocol_error(sc); + } + + sc->last_sid = sid; + sscf = ngx_http_get_module_srv_conf(sc->http_connection->conf_ctx, ngx_http_spdy_module); @@ -968,8 +989,6 @@ ngx_http_spdy_state_syn_stream(ngx_http_ sc->stream = stream; - sc->last_sid = sid; - return ngx_http_spdy_state_headers(sc, pos, end); } From Markus.Linnala at cybercom.com Tue Apr 15 12:26:25 2014 From: Markus.Linnala at cybercom.com (Markus Linnala) Date: Tue, 15 Apr 2014 15:26:25 +0300 Subject: [PATCH 0 of 1] nginx-test proxy_unfinished fix for huge buffers Message-ID: I had problem with new Fedora 20 failing test 15 "no proxy temp". It did not use temp files and so test did not work. To fix this I needed to lower buffer sizes. http://trac.nginx.org/nginx/ticket/541 From Markus.Linnala at cybercom.com Tue Apr 15 12:26:26 2014 From: Markus.Linnala at cybercom.com (Markus Linnala) Date: Tue, 15 Apr 2014 15:26:26 +0300 Subject: [PATCH 1 of 1] Test proxy_unfinished: limit buffers to force buffering on no proxy temp In-Reply-To: References: Message-ID: # HG changeset patch # User Markus Linnala # Date 1397505734 -10800 # Mon Apr 14 23:02:14 2014 +0300 # Node ID cf54213675d59b801bee34328223c28f992cf8c7 # Parent 74a015aad3521ffef6b404cdaa9eafd991824301 Test proxy_unfinished: limit buffers to force buffering on no proxy temp With some systems (Fedora-20 Linux 3.13) socket buffers are huge and nginx does not use temp files when proxying. Set buffer sizes to small values to make use of temp files more probable. Currently there is no way to force temp buffers. no proxy temp should fail on 1.5.2 and work okay on later versions. sub_filter forces chunked encoding and so it is possible to notice problem because last chunk is 0 chunk when there is error. Only use limited buffers on 'no proxy temp' as other tests fail because tests take too long. http://trac.nginx.org/nginx/ticket/541 diff -r 74a015aad352 -r cf54213675d5 proxy_unfinished.t --- a/proxy_unfinished.t Mon Mar 17 16:02:19 2014 +0400 +++ b/proxy_unfinished.t Mon Apr 14 23:02:14 2014 +0300 @@ -19,7 +19,7 @@ use Test::More; -use Socket qw/ CRLF /; +use Socket qw/ CRLF SOL_SOCKET SO_RCVBUF /; BEGIN { use FindBin; chdir($FindBin::Bin); } @@ -33,7 +33,7 @@ plan(skip_all => 'win32') if $^O eq 'MSWin32'; -my $t = Test::Nginx->new()->has(qw/http proxy cache sub/)->plan(15) +my $t = Test::Nginx->new()->has(qw/http proxy cache sub/)->plan(16) ->write_file_expand('nginx.conf', <<'EOF'); %%TEST_GLOBALS%% @@ -72,6 +72,15 @@ add_header X-Cache-Status $upstream_cache_status; } + location /local/ { + alias %%TESTDIR%%/; + } + } + + server { + listen 127.0.0.1:8082 sndbuf=4096; + server_name localhost; + location /proxy/ { sub_filter foo bar; sub_filter_types *; @@ -79,10 +88,6 @@ proxy_buffer_size 1k; proxy_buffers 4 1k; } - - location /local/ { - alias %%TESTDIR%%/; - } } } @@ -159,8 +164,22 @@ # no final chunk chmod(0000, $t->testdir() . '/proxy_temp'); -like(http_get_11('/proxy/big.html', sleep => 0.5), - qr/X(?!.*\x0d\x0a?0\x0d\x0a?)/s, 'no proxy temp'); +# Limit socket buffers here and listen to force buffering to temp file. +like(http_get_11('/proxy/big.html', + socket => + do { + my $s = IO::Socket::INET->new( + Proto => 'tcp', + PeerAddr => '127.0.0.1:8082' + ) + or die "Can't connect to nginx: $!\n"; + $s->setsockopt(SOL_SOCKET, SO_RCVBUF, 1) or die; + $s; + }), + qr/X(?:\x0d\x0a?)?$/s, 'no proxy temp'); + +# If this test fails, previous test did not test buffering properly +is(lines($t, 'error.log', '\[crit\] .*? open\(\) .*? failed .*? request: "GET /proxy/big\.html HTTP/1\.1"'), 1, "no proxy temp actual"); } @@ -270,3 +289,17 @@ } ############################################################################### + +# from error_log.t + +sub lines { + my ($t, $file, $pattern) = @_; + + my $path = $t->testdir() . '/' . $file; + open my $fh, '<', $path or return "$!"; + my $value = map { $_ =~ /$pattern/ } (<$fh>); + close $fh; + return $value; +} + +############################################################################### From mdounin at mdounin.ru Tue Apr 15 13:04:44 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 15 Apr 2014 17:04:44 +0400 Subject: [PATCH 1 of 1] Test proxy_unfinished: limit buffers to force buffering on no proxy temp In-Reply-To: References: Message-ID: <20140415130444.GK34696@mdounin.ru> Hello! On Tue, Apr 15, 2014 at 03:26:26PM +0300, Markus Linnala wrote: > # HG changeset patch > # User Markus Linnala > # Date 1397505734 -10800 > # Mon Apr 14 23:02:14 2014 +0300 > # Node ID cf54213675d59b801bee34328223c28f992cf8c7 > # Parent 74a015aad3521ffef6b404cdaa9eafd991824301 > Test proxy_unfinished: limit buffers to force buffering on no proxy temp > > With some systems (Fedora-20 Linux 3.13) socket buffers are huge and > nginx does not use temp files when proxying. Set buffer sizes to small > values to make use of temp files more probable. Currently there is no > way to force temp buffers. > > no proxy temp should fail on 1.5.2 and work okay on later versions. > > sub_filter forces chunked encoding and so it is possible to notice > problem because last chunk is 0 chunk when there is error. > > Only use limited buffers on 'no proxy temp' as other tests fail > because tests take too long. > > http://trac.nginx.org/nginx/ticket/541 [...] Have you tried something simplier, like --- a/proxy_unfinished.t +++ b/proxy_unfinished.t @@ -50,7 +50,7 @@ http { keys_zone=one:1m; server { - listen 127.0.0.1:8080; + listen 127.0.0.1:8080 sndbuf=4k; server_name localhost; location / { ? -- Maxim Dounin http://nginx.org/ From Markus.Linnala at cybercom.com Tue Apr 15 14:51:25 2014 From: Markus.Linnala at cybercom.com (Markus Linnala) Date: Tue, 15 Apr 2014 17:51:25 +0300 Subject: [PATCH 1 of 1] Test proxy_unfinished: limit buffers to force buffering on no proxy temp In-Reply-To: <20140415130444.GK34696@mdounin.ru> References: <20140415130444.GK34696@mdounin.ru> Message-ID: <534D476D.5070000@cybercom.com> On 15.04.2014 16:04, Maxim Dounin wrote: > Hello! > > On Tue, Apr 15, 2014 at 03:26:26PM +0300, Markus Linnala wrote: > >> # HG changeset patch >> # User Markus Linnala >> # Date 1397505734 -10800 >> # Mon Apr 14 23:02:14 2014 +0300 >> # Node ID cf54213675d59b801bee34328223c28f992cf8c7 >> # Parent 74a015aad3521ffef6b404cdaa9eafd991824301 >> Test proxy_unfinished: limit buffers to force buffering on no proxy temp >> >> With some systems (Fedora-20 Linux 3.13) socket buffers are huge and >> nginx does not use temp files when proxying. Set buffer sizes to small >> values to make use of temp files more probable. Currently there is no >> way to force temp buffers. >> >> no proxy temp should fail on 1.5.2 and work okay on later versions. >> >> sub_filter forces chunked encoding and so it is possible to notice >> problem because last chunk is 0 chunk when there is error. >> >> Only use limited buffers on 'no proxy temp' as other tests fail >> because tests take too long. >> >> http://trac.nginx.org/nginx/ticket/541 > > [...] > > Have you tried something simplier, like > > --- a/proxy_unfinished.t > +++ b/proxy_unfinished.t > @@ -50,7 +50,7 @@ http { > keys_zone=one:1m; > > server { > - listen 127.0.0.1:8080; > + listen 127.0.0.1:8080 sndbuf=4k; > server_name localhost; > > location / { > > > ? > I did not try plain 4k. I did try smaller buffers and 'big (un)?finished un' tests were failing because it took too longer than 5s to do test. Now I ran tests again and if sndbuf is less than 4k or maybe some buffer size you get really bad throughoutput. not ok 13 - big unfinished un # Failed test 'big unfinished un' # at proxy_unfinished.t line 152. # undef # doesn't match '(?^s:unfinished)' not ok 14 - big finished un # Failed test 'big finished un' # at proxy_unfinished.t line 153. # undef # doesn't match '(?^s:finished)' If sndbuf is more than about 36k test seems to fail sometimes. One way to force creating of temp_file is to use proxy_store, but then test does not work as it does not start to response before there is error. With my patch nginx tries to create temp-file when it has sent out 24k of a response. With sndbuf=4k it tries to create temp file around 90k. I see this by running prove under strace. strace -ff -o foo -s 200 -tt prove Extra sleep 0.5s or lowering proxy buffers size does not seem to affect the no proxy temp test. -- Markus Linnala, Chief Systems Architect Cybercom Finland Pakkahuoneenaukio 2 A; 33100 Tampere Mobile +358 40 5919 735 Markus.Linnala at cybercom.com www.cybercom.fi | www.cybercom.com From Markus.Linnala at cybercom.com Tue Apr 15 14:55:51 2014 From: Markus.Linnala at cybercom.com (Markus Linnala) Date: Tue, 15 Apr 2014 17:55:51 +0300 Subject: [PATCH] Test proxy_unfinished: limit sndbuf to force buffering on no proxy temp In-Reply-To: <534D476D.5070000@cybercom.com> References: <534D476D.5070000@cybercom.com> Message-ID: # HG changeset patch # User Markus Linnala # Date 1397505734 -10800 # Mon Apr 14 23:02:14 2014 +0300 # Node ID a548e5a5cf251ee97fb782d6b9e6c374389782ea # Parent 74a015aad3521ffef6b404cdaa9eafd991824301 Test proxy_unfinished: limit sndbuf to force buffering on no proxy temp With some systems (Fedora-20 Linux 3.13) socket buffers are huge and nginx does not use temp files when proxying. Set sndbuf to small values to make use of temp files around 90k sent. Currently there is no way to force temp buffers. If sndbuf is more than 36k test seems to fail. no proxy temp should fail on 1.5.2 and work okay on later versions. sub_filter forces chunked encoding and so it is possible to notice problem because last chunk is 0 chunk when there is error. http://trac.nginx.org/nginx/ticket/541 I did not notice any changes with sleep or limited proxy buffers. So I dropped them. Changed patch as per suggestion from Maxim Dounin. diff -r 74a015aad352 -r a548e5a5cf25 proxy_unfinished.t --- a/proxy_unfinished.t Mon Mar 17 16:02:19 2014 +0400 +++ b/proxy_unfinished.t Mon Apr 14 23:02:14 2014 +0300 @@ -50,7 +50,7 @@ keys_zone=one:1m; server { - listen 127.0.0.1:8080; + listen 127.0.0.1:8080 sndbuf=4k; server_name localhost; location / { @@ -76,8 +76,6 @@ sub_filter foo bar; sub_filter_types *; proxy_pass http://127.0.0.1:8080/local/; - proxy_buffer_size 1k; - proxy_buffers 4 1k; } location /local/ { @@ -159,7 +157,7 @@ # no final chunk chmod(0000, $t->testdir() . '/proxy_temp'); -like(http_get_11('/proxy/big.html', sleep => 0.5), +like(http_get_11('/proxy/big.html'), qr/X(?!.*\x0d\x0a?0\x0d\x0a?)/s, 'no proxy temp'); } From vl at nginx.com Tue Apr 15 17:57:24 2014 From: vl at nginx.com (Homutov Vladimir) Date: Tue, 15 Apr 2014 21:57:24 +0400 Subject: [PATCH 1 of 1] Test proxy_unfinished: limit buffers to force buffering on no proxy temp In-Reply-To: References: Message-ID: <20140415175723.GA8289@vl> On Tue, Apr 15, 2014 at 03:26:26PM +0300, Markus Linnala wrote: > > With some systems (Fedora-20 Linux 3.13) socket buffers are huge and Can you please share exact values? Are they default or you tuned it to meet some of your needs? (/proc/sys/net/ipv4/tcp_rmem, /proc/sys/net/ipv4/tcp_wmem) From mdounin at mdounin.ru Tue Apr 15 18:07:06 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 15 Apr 2014 22:07:06 +0400 Subject: [PATCH 1 of 1] Test proxy_unfinished: limit buffers to force buffering on no proxy temp In-Reply-To: <534D476D.5070000@cybercom.com> References: <20140415130444.GK34696@mdounin.ru> <534D476D.5070000@cybercom.com> Message-ID: <20140415180706.GQ34696@mdounin.ru> Hello! On Tue, Apr 15, 2014 at 05:51:25PM +0300, Markus Linnala wrote: > On 15.04.2014 16:04, Maxim Dounin wrote: > >Hello! > > > >On Tue, Apr 15, 2014 at 03:26:26PM +0300, Markus Linnala wrote: > > > >># HG changeset patch > >># User Markus Linnala > >># Date 1397505734 -10800 > >># Mon Apr 14 23:02:14 2014 +0300 > >># Node ID cf54213675d59b801bee34328223c28f992cf8c7 > >># Parent 74a015aad3521ffef6b404cdaa9eafd991824301 > >>Test proxy_unfinished: limit buffers to force buffering on no proxy temp > >> > >>With some systems (Fedora-20 Linux 3.13) socket buffers are huge and > >>nginx does not use temp files when proxying. Set buffer sizes to small > >>values to make use of temp files more probable. Currently there is no > >>way to force temp buffers. > >> > >>no proxy temp should fail on 1.5.2 and work okay on later versions. > >> > >>sub_filter forces chunked encoding and so it is possible to notice > >>problem because last chunk is 0 chunk when there is error. > >> > >>Only use limited buffers on 'no proxy temp' as other tests fail > >>because tests take too long. > >> > >>http://trac.nginx.org/nginx/ticket/541 > > > >[...] > > > >Have you tried something simplier, like > > > >--- a/proxy_unfinished.t > >+++ b/proxy_unfinished.t > >@@ -50,7 +50,7 @@ http { > > keys_zone=one:1m; > > > > server { > >- listen 127.0.0.1:8080; > >+ listen 127.0.0.1:8080 sndbuf=4k; > > server_name localhost; > > > > location / { > > > > > >? > > > > I did not try plain 4k. > > I did try smaller buffers and 'big (un)?finished un' tests were failing > because it took too longer than 5s to do test. Now I ran tests again and if > sndbuf is less than 4k or maybe some buffer size you get really bad > throughoutput. > > not ok 13 - big unfinished un > > # Failed test 'big unfinished un' > # at proxy_unfinished.t line 152. > # undef > # doesn't match '(?^s:unfinished)' > not ok 14 - big finished un > > # Failed test 'big finished un' > # at proxy_unfinished.t line 153. > # undef > # doesn't match '(?^s:finished)' > > > If sndbuf is more than about 36k test seems to fail sometimes. > > One way to force creating of temp_file is to use proxy_store, but then test > does not work as it does not start to response before there is error. > > With my patch nginx tries to create temp-file when it has sent out 24k of a > response. With sndbuf=4k it tries to create temp file around 90k. I see this > by running prove under strace. > > strace -ff -o foo -s 200 -tt prove > > Extra sleep 0.5s or lowering proxy buffers size does not seem to affect the > no proxy temp test. There are reasons for both sleep and low proxy buffers: - sleep is to make sure that client isn't reading too fast to make buffering unneeded; - low proxy buffers are to reduce buffering in nginx itself. I've committed a patch with sndbuf=32k and a regular expression change to silently allow full responses. http://hg.nginx.org/nginx-tests/rev/915ef26ac6eb -- Maxim Dounin http://nginx.org/ From piotr at cloudflare.com Tue Apr 15 19:44:37 2014 From: piotr at cloudflare.com (Piotr Sikora) Date: Tue, 15 Apr 2014 12:44:37 -0700 Subject: [PATCH] SSL: support automatic selection of ECDH temporary key parameters In-Reply-To: References: <20140401170037.GZ34696@mdounin.ru> Message-ID: Hey Maxim, >> - If nginx was compiled with OpenSSL 1.0.2, but used with an >> older version, things will not work at all; this is not something >> completely unacceptable, but it's something we may want to >> avoid. > > Will look into it. How about adding check to make sure that OpenSSL version nginx was built against (i.e. version info from the headers) matches the version from the library we're loading (i.e. version info from the runtime)? Best regards, Piotr Sikora From vbart at nginx.com Wed Apr 16 07:41:18 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Wed, 16 Apr 2014 07:41:18 +0000 Subject: [nginx] Fixed missing "static" in declaration of ngx_http_gzip_q... Message-ID: details: http://hg.nginx.org/nginx/rev/b51bd021808b branches: changeset: 5651:b51bd021808b user: Valentin Bartenev date: Wed Apr 16 11:40:38 2014 +0400 description: Fixed missing "static" in declaration of ngx_http_gzip_quantity(). diffstat: src/http/ngx_http_core_module.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r 363395795384 -r b51bd021808b src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c Wed Apr 09 18:15:32 2014 +0400 +++ b/src/http/ngx_http_core_module.c Wed Apr 16 11:40:38 2014 +0400 @@ -2368,7 +2368,7 @@ equal: } -ngx_uint_t +static ngx_uint_t ngx_http_gzip_quantity(u_char *p, u_char *last) { u_char c; From vbart at nginx.com Wed Apr 16 07:41:20 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Wed, 16 Apr 2014 07:41:20 +0000 Subject: [nginx] SPDY: fixed typo in log message. Message-ID: details: http://hg.nginx.org/nginx/rev/b6240baead00 branches: changeset: 5652:b6240baead00 user: Valentin Bartenev date: Wed Apr 16 11:40:42 2014 +0400 description: SPDY: fixed typo in log message. diffstat: src/http/ngx_http_spdy.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r b51bd021808b -r b6240baead00 src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Wed Apr 16 11:40:38 2014 +0400 +++ b/src/http/ngx_http_spdy.c Wed Apr 16 11:40:42 2014 +0400 @@ -943,7 +943,7 @@ ngx_http_spdy_state_syn_stream(ngx_http_ if (sc->processing >= sscf->concurrent_streams) { ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, - "spdy concurrent streams excessed %ui", sc->processing); + "spdy concurrent streams exceeded %ui", sc->processing); if (ngx_http_spdy_send_rst_stream(sc, sid, NGX_SPDY_REFUSED_STREAM, prio) From mdounin at mdounin.ru Wed Apr 16 10:00:13 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 16 Apr 2014 14:00:13 +0400 Subject: [PATCH] SSL: support automatic selection of ECDH temporary key parameters In-Reply-To: References: <20140401170037.GZ34696@mdounin.ru> Message-ID: <20140416100013.GY34696@mdounin.ru> Hello! On Tue, Apr 15, 2014 at 12:44:37PM -0700, Piotr Sikora wrote: > Hey Maxim, > > >> - If nginx was compiled with OpenSSL 1.0.2, but used with an > >> older version, things will not work at all; this is not something > >> completely unacceptable, but it's something we may want to > >> avoid. > > > > Will look into it. > > How about adding check to make sure that OpenSSL version nginx was > built against (i.e. version info from the headers) matches the version > from the library we're loading (i.e. version info from the runtime)? I don't think check per se is a good idea - in particular, nginx should be able to start with any newer version of OpenSSL. If there is no easy solution (like, e.g., with SNI, where we check SSL_CTX_set_tlsext_servername_callback() result and act accordingly) - there is no need to bother. -- Maxim Dounin http://nginx.org/ From phoem at phoempc.com Thu Apr 17 16:15:54 2014 From: phoem at phoempc.com (Jordan Newman) Date: Thu, 17 Apr 2014 12:15:54 -0400 Subject: Tracking socket writes Message-ID: <8461A4FF-D563-4FD0-85EB-6AB5764177B8@phoempc.com> Hello, I was hoping to be pointed in the right direction, I am hoping to write a module that will need to get a notification after every write() to a socket has been completed. This includes aio_write() sendfile() etc. There does not appear to be a connect/accept phase that I can plug into. I was planning on setting up a write event and then doing math to see how much data has been written to the socket to fill it each time but I am open to other suggestions as well. Any advice you can provide would be greatly appreciated. Thanks in advance for your help. -- Jordan Newman From aaron.peschel at gmail.com Fri Apr 18 00:39:40 2014 From: aaron.peschel at gmail.com (Aaron Peschel) Date: Thu, 17 Apr 2014 17:39:40 -0700 Subject: [nginx] Adding Support for Weak ETags Message-ID: Hello, I am interested in getting support for Weak ETags into the mainline. There was some discussion previously in here previously that developed a quick patch to add support. What additional functionality would be required and what steps should be followed to get weak etag functionality added to nginx? I am willing to do the work, I just need some help with heading in the right direction. Thank you, -Aaron Peschel From pluknet at nginx.com Fri Apr 18 08:32:42 2014 From: pluknet at nginx.com (Sergey Kandaurov) Date: Fri, 18 Apr 2014 08:32:42 +0000 Subject: [nginx] Access log: the "if" parameter of the "access_log" direc... Message-ID: details: http://hg.nginx.org/nginx/rev/cb308813b453 branches: changeset: 5653:cb308813b453 user: Sergey Kandaurov date: Tue Apr 15 21:32:56 2014 +0400 description: Access log: the "if" parameter of the "access_log" directive. The parameter value specifies a condition under which the request is logged. diffstat: src/http/modules/ngx_http_log_module.c | 57 ++++++++++++++++++++++++++++----- 1 files changed, 47 insertions(+), 10 deletions(-) diffs (109 lines): diff -r b6240baead00 -r cb308813b453 src/http/modules/ngx_http_log_module.c --- a/src/http/modules/ngx_http_log_module.c Wed Apr 16 11:40:42 2014 +0400 +++ b/src/http/modules/ngx_http_log_module.c Tue Apr 15 21:32:56 2014 +0400 @@ -67,6 +67,7 @@ typedef struct { time_t disk_full_time; time_t error_log_time; ngx_http_log_fmt_t *format; + ngx_http_complex_value_t *filter; } ngx_http_log_t; @@ -240,6 +241,7 @@ ngx_http_log_handler(ngx_http_request_t { u_char *line, *p; size_t len; + ngx_str_t val; ngx_uint_t i, l; ngx_http_log_t *log; ngx_http_log_op_t *op; @@ -258,6 +260,16 @@ ngx_http_log_handler(ngx_http_request_t log = lcf->logs->elts; for (l = 0; l < lcf->logs->nelts; l++) { + if (log[l].filter) { + if (ngx_http_complex_value(r, log[l].filter, &val) != NGX_OK) { + return NGX_ERROR; + } + + if (val.len == 0 || (val.len == 1 && val.data[0] == '0')) { + continue; + } + } + if (ngx_time() == log[l].disk_full_time) { /* @@ -1085,16 +1097,17 @@ ngx_http_log_set_log(ngx_conf_t *cf, ngx { ngx_http_log_loc_conf_t *llcf = conf; - ssize_t size; - ngx_int_t gzip; - ngx_uint_t i, n; - ngx_msec_t flush; - ngx_str_t *value, name, s; - ngx_http_log_t *log; - ngx_http_log_buf_t *buffer; - ngx_http_log_fmt_t *fmt; - ngx_http_log_main_conf_t *lmcf; - ngx_http_script_compile_t sc; + ssize_t size; + ngx_int_t gzip; + ngx_uint_t i, n; + ngx_msec_t flush; + ngx_str_t *value, name, s, filter; + ngx_http_log_t *log; + ngx_http_log_buf_t *buffer; + ngx_http_log_fmt_t *fmt; + ngx_http_log_main_conf_t *lmcf; + ngx_http_script_compile_t sc; + ngx_http_compile_complex_value_t ccv; value = cf->args->elts; @@ -1189,6 +1202,7 @@ ngx_http_log_set_log(ngx_conf_t *cf, ngx size = 0; flush = 0; gzip = 0; + filter.len = 0; for (i = 3; i < cf->args->nelts; i++) { @@ -1255,6 +1269,12 @@ ngx_http_log_set_log(ngx_conf_t *cf, ngx #endif } + if (ngx_strncmp(value[i].data, "if=", 3) == 0) { + filter.len = value[i].len - 3; + filter.data = value[i].data + 3; + continue; + } + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "invalid parameter \"%V\"", &value[i]); return NGX_CONF_ERROR; @@ -1324,6 +1344,23 @@ ngx_http_log_set_log(ngx_conf_t *cf, ngx log->file->data = buffer; } + if (filter.len) { + log->filter = ngx_palloc(cf->pool, sizeof(ngx_http_complex_value_t)); + if (log->filter == NULL) { + return NGX_CONF_ERROR; + } + + ngx_memzero(&ccv, sizeof(ngx_http_compile_complex_value_t)); + + ccv.cf = cf; + ccv.value = &filter; + ccv.complex_value = log->filter; + + if (ngx_http_compile_complex_value(&ccv) != NGX_OK) { + return NGX_CONF_ERROR; + } + } + return NGX_CONF_OK; } From mdounin at mdounin.ru Fri Apr 18 16:33:31 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 18 Apr 2014 16:33:31 +0000 Subject: [nginx] stable-1.6 branch Message-ID: details: http://hg.nginx.org/nginx/rev/1dfb75c43080 branches: stable-1.6 changeset: 5654:1dfb75c43080 user: Maxim Dounin date: Fri Apr 18 20:08:47 2014 +0400 description: stable-1.6 branch From mdounin at mdounin.ru Fri Apr 18 16:33:33 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 18 Apr 2014 16:33:33 +0000 Subject: [nginx] Version bump. Message-ID: details: http://hg.nginx.org/nginx/rev/0373d5141fa7 branches: stable-1.6 changeset: 5655:0373d5141fa7 user: Maxim Dounin date: Fri Apr 18 20:10:42 2014 +0400 description: Version bump. diffstat: src/core/nginx.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (14 lines): diff --git a/src/core/nginx.h b/src/core/nginx.h --- a/src/core/nginx.h +++ b/src/core/nginx.h @@ -9,8 +9,8 @@ #define _NGINX_H_INCLUDED_ -#define nginx_version 1005014 -#define NGINX_VERSION "1.5.14" +#define nginx_version 1006000 +#define NGINX_VERSION "1.6.0" #define NGINX_VER "nginx/" NGINX_VERSION #define NGINX_VAR "NGINX" From mdounin at mdounin.ru Fri Apr 18 16:33:34 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 18 Apr 2014 16:33:34 +0000 Subject: [nginx] Version bump. Message-ID: details: http://hg.nginx.org/nginx/rev/3838a3d4bf79 branches: changeset: 5656:3838a3d4bf79 user: Maxim Dounin date: Fri Apr 18 20:12:53 2014 +0400 description: Version bump. diffstat: src/core/nginx.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (14 lines): diff --git a/src/core/nginx.h b/src/core/nginx.h --- a/src/core/nginx.h +++ b/src/core/nginx.h @@ -9,8 +9,8 @@ #define _NGINX_H_INCLUDED_ -#define nginx_version 1005014 -#define NGINX_VERSION "1.5.14" +#define nginx_version 1007000 +#define NGINX_VERSION "1.7.0" #define NGINX_VER "nginx/" NGINX_VERSION #define NGINX_VAR "NGINX" From mdounin at mdounin.ru Fri Apr 18 16:33:36 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 18 Apr 2014 16:33:36 +0000 Subject: [nginx] SSL: fixed misuse of NGX_LOG_DEBUG_HTTP. Message-ID: details: http://hg.nginx.org/nginx/rev/3b48f9e69e70 branches: changeset: 5657:3b48f9e69e70 user: Maxim Dounin date: Fri Apr 18 20:13:14 2014 +0400 description: SSL: fixed misuse of NGX_LOG_DEBUG_HTTP. diffstat: src/event/ngx_event_openssl.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diffs (30 lines): diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c --- a/src/event/ngx_event_openssl.c +++ b/src/event/ngx_event_openssl.c @@ -2418,7 +2418,7 @@ ngx_ssl_session_ticket_key_callback(ngx_ if (enc == 1) { /* encrypt session ticket */ - ngx_log_debug3(NGX_LOG_DEBUG_HTTP, c->log, 0, + ngx_log_debug3(NGX_LOG_DEBUG_EVENT, c->log, 0, "ssl session ticket encrypt, key: \"%*s\" (%s session)", ngx_hex_dump(buf, key[0].name, 16) - buf, buf, SSL_session_reused(ssl_conn) ? "reused" : "new"); @@ -2440,7 +2440,7 @@ ngx_ssl_session_ticket_key_callback(ngx_ } } - ngx_log_debug2(NGX_LOG_DEBUG_HTTP, c->log, 0, + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0, "ssl session ticket decrypt, key: \"%*s\" not found", ngx_hex_dump(buf, name, 16) - buf, buf); @@ -2448,7 +2448,7 @@ ngx_ssl_session_ticket_key_callback(ngx_ found: - ngx_log_debug3(NGX_LOG_DEBUG_HTTP, c->log, 0, + ngx_log_debug3(NGX_LOG_DEBUG_EVENT, c->log, 0, "ssl session ticket decrypt, key: \"%*s\"%s", ngx_hex_dump(buf, key[i].name, 16) - buf, buf, (i == 0) ? " (default)" : ""); From mdounin at mdounin.ru Fri Apr 18 16:33:37 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 18 Apr 2014 16:33:37 +0000 Subject: [nginx] SSL: $ssl_server_name variable. Message-ID: details: http://hg.nginx.org/nginx/rev/94ae92776441 branches: changeset: 5658:94ae92776441 user: Maxim Dounin date: Fri Apr 18 20:13:21 2014 +0400 description: SSL: $ssl_server_name variable. diffstat: src/event/ngx_event_openssl.c | 22 ++++++++++++++++++++++ src/event/ngx_event_openssl.h | 2 ++ src/http/modules/ngx_http_ssl_module.c | 3 +++ 3 files changed, 27 insertions(+), 0 deletions(-) diffs (57 lines): diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c --- a/src/event/ngx_event_openssl.c +++ b/src/event/ngx_event_openssl.c @@ -2545,6 +2545,28 @@ ngx_ssl_get_session_reused(ngx_connectio ngx_int_t +ngx_ssl_get_server_name(ngx_connection_t *c, ngx_pool_t *pool, ngx_str_t *s) +{ +#ifdef SSL_CTRL_SET_TLSEXT_HOSTNAME + + const char *servername; + + servername = SSL_get_servername(c->ssl->connection, + TLSEXT_NAMETYPE_host_name); + if (servername) { + s->data = (u_char *) servername; + s->len = ngx_strlen(servername); + return NGX_OK; + } + +#endif + + s->len = 0; + return NGX_OK; +} + + +ngx_int_t ngx_ssl_get_raw_certificate(ngx_connection_t *c, ngx_pool_t *pool, ngx_str_t *s) { size_t len; diff --git a/src/event/ngx_event_openssl.h b/src/event/ngx_event_openssl.h --- a/src/event/ngx_event_openssl.h +++ b/src/event/ngx_event_openssl.h @@ -159,6 +159,8 @@ ngx_int_t ngx_ssl_get_session_id(ngx_con ngx_str_t *s); ngx_int_t ngx_ssl_get_session_reused(ngx_connection_t *c, ngx_pool_t *pool, ngx_str_t *s); +ngx_int_t ngx_ssl_get_server_name(ngx_connection_t *c, ngx_pool_t *pool, + ngx_str_t *s); ngx_int_t ngx_ssl_get_raw_certificate(ngx_connection_t *c, ngx_pool_t *pool, ngx_str_t *s); ngx_int_t ngx_ssl_get_certificate(ngx_connection_t *c, ngx_pool_t *pool, diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c --- a/src/http/modules/ngx_http_ssl_module.c +++ b/src/http/modules/ngx_http_ssl_module.c @@ -273,6 +273,9 @@ static ngx_http_variable_t ngx_http_ssl { ngx_string("ssl_session_reused"), NULL, ngx_http_ssl_variable, (uintptr_t) ngx_ssl_get_session_reused, NGX_HTTP_VAR_CHANGEABLE, 0 }, + { ngx_string("ssl_server_name"), NULL, ngx_http_ssl_variable, + (uintptr_t) ngx_ssl_get_server_name, NGX_HTTP_VAR_CHANGEABLE, 0 }, + { ngx_string("ssl_client_cert"), NULL, ngx_http_ssl_variable, (uintptr_t) ngx_ssl_get_certificate, NGX_HTTP_VAR_CHANGEABLE, 0 }, From mdounin at mdounin.ru Fri Apr 18 16:33:38 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 18 Apr 2014 16:33:38 +0000 Subject: [nginx] Upstream: plugged potential memory leak on reload. Message-ID: details: http://hg.nginx.org/nginx/rev/3fb6615bb87f branches: changeset: 5659:3fb6615bb87f user: Maxim Dounin date: Fri Apr 18 20:13:24 2014 +0400 description: Upstream: plugged potential memory leak on reload. The SSL_CTX_set_cipher_list() may fail if there are no valid ciphers specified in proxy_ssl_ciphers / uwsgi_ssl_ciphers, resulting in SSL context leak. In theory, ngx_pool_cleanup_add() may fail too, but this case is intentionally left out for now as it's almost impossible and proper fix will require changes to http ssl and mail ssl code as well. diffstat: src/http/modules/ngx_http_proxy_module.c | 16 ++++++++-------- src/http/modules/ngx_http_uwsgi_module.c | 16 ++++++++-------- 2 files changed, 16 insertions(+), 16 deletions(-) diffs (66 lines): diff --git a/src/http/modules/ngx_http_proxy_module.c b/src/http/modules/ngx_http_proxy_module.c --- a/src/http/modules/ngx_http_proxy_module.c +++ b/src/http/modules/ngx_http_proxy_module.c @@ -3774,6 +3774,14 @@ ngx_http_proxy_set_ssl(ngx_conf_t *cf, n return NGX_ERROR; } + cln = ngx_pool_cleanup_add(cf->pool, 0); + if (cln == NULL) { + return NGX_ERROR; + } + + cln->handler = ngx_ssl_cleanup_ctx; + cln->data = plcf->upstream.ssl; + if (SSL_CTX_set_cipher_list(plcf->upstream.ssl->ctx, (const char *) plcf->ssl_ciphers.data) == 0) @@ -3784,14 +3792,6 @@ ngx_http_proxy_set_ssl(ngx_conf_t *cf, n return NGX_ERROR; } - cln = ngx_pool_cleanup_add(cf->pool, 0); - if (cln == NULL) { - return NGX_ERROR; - } - - cln->handler = ngx_ssl_cleanup_ctx; - cln->data = plcf->upstream.ssl; - return NGX_OK; } diff --git a/src/http/modules/ngx_http_uwsgi_module.c b/src/http/modules/ngx_http_uwsgi_module.c --- a/src/http/modules/ngx_http_uwsgi_module.c +++ b/src/http/modules/ngx_http_uwsgi_module.c @@ -2012,6 +2012,14 @@ ngx_http_uwsgi_set_ssl(ngx_conf_t *cf, n return NGX_ERROR; } + cln = ngx_pool_cleanup_add(cf->pool, 0); + if (cln == NULL) { + return NGX_ERROR; + } + + cln->handler = ngx_ssl_cleanup_ctx; + cln->data = uwcf->upstream.ssl; + if (SSL_CTX_set_cipher_list(uwcf->upstream.ssl->ctx, (const char *) uwcf->ssl_ciphers.data) == 0) @@ -2022,14 +2030,6 @@ ngx_http_uwsgi_set_ssl(ngx_conf_t *cf, n return NGX_ERROR; } - cln = ngx_pool_cleanup_add(cf->pool, 0); - if (cln == NULL) { - return NGX_ERROR; - } - - cln->handler = ngx_ssl_cleanup_ctx; - cln->data = uwcf->upstream.ssl; - return NGX_OK; } From mdounin at mdounin.ru Fri Apr 18 16:33:40 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 18 Apr 2014 16:33:40 +0000 Subject: [nginx] Upstream: proxy_ssl_name and proxy_ssl_server_name direc... Message-ID: details: http://hg.nginx.org/nginx/rev/7022564a9e0e branches: changeset: 5660:7022564a9e0e user: Maxim Dounin date: Fri Apr 18 20:13:28 2014 +0400 description: Upstream: proxy_ssl_name and proxy_ssl_server_name directives. These directives allow to switch on Server Name Indication (SNI) while connecting to upstream servers. By default, proxy_ssl_server_name is currently off (that is, no SNI) and proxy_ssl_name is set to a host used in the proxy_pass directive. diffstat: src/http/modules/ngx_http_proxy_module.c | 26 +++++++ src/http/ngx_http_upstream.c | 104 +++++++++++++++++++++++++++++++ src/http/ngx_http_upstream.h | 7 ++ 3 files changed, 137 insertions(+), 0 deletions(-) diffs (231 lines): diff --git a/src/http/modules/ngx_http_proxy_module.c b/src/http/modules/ngx_http_proxy_module.c --- a/src/http/modules/ngx_http_proxy_module.c +++ b/src/http/modules/ngx_http_proxy_module.c @@ -553,6 +553,20 @@ static ngx_command_t ngx_http_proxy_com offsetof(ngx_http_proxy_loc_conf_t, ssl_ciphers), NULL }, + { ngx_string("proxy_ssl_name"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, + ngx_http_set_complex_value_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_proxy_loc_conf_t, upstream.ssl_name), + NULL }, + + { ngx_string("proxy_ssl_server_name"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_proxy_loc_conf_t, upstream.ssl_server_name), + NULL }, + #endif ngx_null_command @@ -2390,6 +2404,7 @@ ngx_http_proxy_create_loc_conf(ngx_conf_ * conf->upstream.location = NULL; * conf->upstream.store_lengths = NULL; * conf->upstream.store_values = NULL; + * conf->upstream.ssl_name = NULL; * * conf->method = { 0, NULL }; * conf->headers_source = NULL; @@ -2441,8 +2456,10 @@ ngx_http_proxy_create_loc_conf(ngx_conf_ conf->upstream.pass_headers = NGX_CONF_UNSET_PTR; conf->upstream.intercept_errors = NGX_CONF_UNSET; + #if (NGX_HTTP_SSL) conf->upstream.ssl_session_reuse = NGX_CONF_UNSET; + conf->upstream.ssl_server_name = NGX_CONF_UNSET; #endif /* "proxy_cyclic_temp_file" is disabled */ @@ -2714,6 +2731,7 @@ ngx_http_proxy_merge_loc_conf(ngx_conf_t prev->upstream.intercept_errors, 0); #if (NGX_HTTP_SSL) + ngx_conf_merge_value(conf->upstream.ssl_session_reuse, prev->upstream.ssl_session_reuse, 1); @@ -2725,9 +2743,17 @@ ngx_http_proxy_merge_loc_conf(ngx_conf_t ngx_conf_merge_str_value(conf->ssl_ciphers, prev->ssl_ciphers, "DEFAULT"); + if (conf->upstream.ssl_name == NULL) { + conf->upstream.ssl_name = prev->upstream.ssl_name; + } + + ngx_conf_merge_value(conf->upstream.ssl_server_name, + prev->upstream.ssl_server_name, 0); + if (conf->ssl && ngx_http_proxy_set_ssl(cf, conf) != NGX_OK) { return NGX_CONF_ERROR; } + #endif ngx_conf_merge_value(conf->redirect, prev->redirect, 1); diff --git a/src/http/ngx_http_upstream.c b/src/http/ngx_http_upstream.c --- a/src/http/ngx_http_upstream.c +++ b/src/http/ngx_http_upstream.c @@ -156,6 +156,8 @@ static char *ngx_http_upstream_init_main static void ngx_http_upstream_ssl_init_connection(ngx_http_request_t *, ngx_http_upstream_t *u, ngx_connection_t *c); static void ngx_http_upstream_ssl_handshake(ngx_connection_t *c); +static ngx_int_t ngx_http_upstream_ssl_name(ngx_http_request_t *r, + ngx_http_upstream_t *u, ngx_connection_t *c); #endif @@ -581,9 +583,16 @@ ngx_http_upstream_init_request(ngx_http_ if (u->resolved == NULL) { uscf = u->conf->upstream; +#if (NGX_HTTP_SSL) + u->ssl_name = uscf->host; +#endif } else { +#if (NGX_HTTP_SSL) + u->ssl_name = u->resolved->host; +#endif + if (u->resolved->sockaddr) { if (ngx_http_upstream_create_round_robin_peer(r, u->resolved) @@ -1355,6 +1364,14 @@ ngx_http_upstream_ssl_init_connection(ng c->sendfile = 0; u->output.sendfile = 0; + if (u->conf->ssl_server_name) { + if (ngx_http_upstream_ssl_name(r, u, c) != NGX_OK) { + ngx_http_upstream_finalize_request(r, u, + NGX_HTTP_INTERNAL_SERVER_ERROR); + return; + } + } + if (u->conf->ssl_session_reuse) { if (u->peer.set_session(&u->peer, u->peer.data) != NGX_OK) { ngx_http_upstream_finalize_request(r, u, @@ -1409,6 +1426,93 @@ ngx_http_upstream_ssl_handshake(ngx_conn ngx_http_run_posted_requests(c); } + +static ngx_int_t +ngx_http_upstream_ssl_name(ngx_http_request_t *r, ngx_http_upstream_t *u, + ngx_connection_t *c) +{ + u_char *p, *last; + ngx_str_t name; + + if (u->conf->ssl_name) { + if (ngx_http_complex_value(r, u->conf->ssl_name, &name) != NGX_OK) { + return NGX_ERROR; + } + + } else { + name = u->ssl_name; + } + + if (name.len == 0) { + goto done; + } + + /* + * ssl name here may contain port, notably if derived from $proxy_host + * or $http_host; we have to strip it + */ + + p = name.data; + last = name.data + name.len; + + if (*p == '[') { + p = ngx_strlchr(p, last, ']'); + + if (p == NULL) { + p = name.data; + } + } + + p = ngx_strlchr(p, last, ':'); + + if (p != NULL) { + name.len = p - name.data; + } + +#ifdef SSL_CTRL_SET_TLSEXT_HOSTNAME + + /* as per RFC 6066, literal IPv4 and IPv6 addresses are not permitted */ + + if (name.len == 0 || *name.data == '[') { + goto done; + } + + if (ngx_inet_addr(name.data, name.len) != INADDR_NONE) { + goto done; + } + + /* + * SSL_set_tlsext_host_name() needs a null-terminated string, + * hence we explicitly null-terminate name here + */ + + p = ngx_palloc(r->pool, name.len + 1); + if (p == NULL) { + return NGX_ERROR; + } + + (void) ngx_cpystrn(p, name.data, name.len + 1); + + name.data = p; + + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "upstream SSL server name: \"%s\"", name.data); + + if (SSL_set_tlsext_host_name(c->ssl->connection, name.data) == 0) { + ngx_ssl_error(NGX_LOG_ERR, r->connection->log, 0, + "SSL_set_tlsext_host_name(\"%s\") failed", name.data); + return NGX_ERROR; + } + +#endif + +done: + + u->ssl_name = name; + + return NGX_OK; +} + #endif diff --git a/src/http/ngx_http_upstream.h b/src/http/ngx_http_upstream.h --- a/src/http/ngx_http_upstream.h +++ b/src/http/ngx_http_upstream.h @@ -195,6 +195,9 @@ typedef struct { #if (NGX_HTTP_SSL) ngx_ssl_t *ssl; ngx_flag_t ssl_session_reuse; + + ngx_http_complex_value_t *ssl_name; + ngx_flag_t ssl_server_name; #endif ngx_str_t module; @@ -323,6 +326,10 @@ struct ngx_http_upstream_s { ngx_str_t schema; ngx_str_t uri; +#if (NGX_HTTP_SSL) + ngx_str_t ssl_name; +#endif + ngx_http_cleanup_pt *cleanup; unsigned store:1; From mdounin at mdounin.ru Fri Apr 18 16:33:41 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 18 Apr 2014 16:33:41 +0000 Subject: [nginx] Upstream: proxy_ssl_verify and friends. Message-ID: details: http://hg.nginx.org/nginx/rev/060c2e692b96 branches: changeset: 5661:060c2e692b96 user: Maxim Dounin date: Fri Apr 18 20:13:30 2014 +0400 description: Upstream: proxy_ssl_verify and friends. diffstat: src/event/ngx_event_openssl.c | 164 +++++++++++++++++++++++++++++++ src/event/ngx_event_openssl.h | 2 + src/http/modules/ngx_http_proxy_module.c | 62 +++++++++++ src/http/ngx_http_upstream.c | 27 ++++- src/http/ngx_http_upstream.h | 1 + 5 files changed, 255 insertions(+), 1 deletions(-) diffs (truncated from 376 to 300 lines): diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c --- a/src/event/ngx_event_openssl.c +++ b/src/event/ngx_event_openssl.c @@ -44,6 +44,10 @@ static int ngx_ssl_session_ticket_key_ca HMAC_CTX *hctx, int enc); #endif +#if OPENSSL_VERSION_NUMBER < 0x10002001L +static ngx_int_t ngx_ssl_check_name(ngx_str_t *name, ASN1_STRING *str); +#endif + static void *ngx_openssl_create_conf(ngx_cycle_t *cycle); static char *ngx_openssl_engine(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); static void ngx_openssl_exit(ngx_cycle_t *cycle); @@ -2487,6 +2491,166 @@ ngx_ssl_cleanup_ctx(void *data) ngx_int_t +ngx_ssl_check_host(ngx_connection_t *c, ngx_str_t *name) +{ + X509 *cert; + + cert = SSL_get_peer_certificate(c->ssl->connection); + if (cert == NULL) { + return NGX_ERROR; + } + +#if OPENSSL_VERSION_NUMBER >= 0x10002001L + + /* X509_check_host() is only available in OpenSSL 1.0.2+ */ + + if (X509_check_host(cert, name->data, name->len, 0) != 1) { + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, + "X509_check_host(): no match"); + goto failed; + } + + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, + "X509_check_host(): match"); + + goto found; + +#else + { + int n, i; + X509_NAME *sname; + ASN1_STRING *str; + X509_NAME_ENTRY *entry; + GENERAL_NAME *altname; + STACK_OF(GENERAL_NAME) *altnames; + + /* + * As per RFC6125 and RFC2818, we check subjectAltName extension, + * and if it's not present - commonName in Subject is checked. + */ + + altnames = X509_get_ext_d2i(cert, NID_subject_alt_name, NULL, NULL); + + if (altnames) { + n = sk_GENERAL_NAME_num(altnames); + + for (i = 0; i < n; i++) { + altname = sk_GENERAL_NAME_value(altnames, i); + + if (altname->type != GEN_DNS) { + continue; + } + + str = altname->d.dNSName; + + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0, + "SSL subjectAltName: \"%*s\"", + ASN1_STRING_length(str), ASN1_STRING_data(str)); + + if (ngx_ssl_check_name(name, str) == NGX_OK) { + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, + "SSL subjectAltName: match"); + GENERAL_NAMES_free(altnames); + goto found; + } + } + + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, + "SSL subjectAltName: no match"); + + GENERAL_NAMES_free(altnames); + goto failed; + } + + /* + * If there is no subjectAltName extension, check commonName + * in Subject. While RFC2818 requires to only check "most specific" + * CN, both Apache and OpenSSL check all CNs, and so do we. + */ + + sname = X509_get_subject_name(cert); + + if (sname == NULL) { + goto failed; + } + + i = -1; + for ( ;; ) { + i = X509_NAME_get_index_by_NID(sname, NID_commonName, i); + + if (i < 0) { + break; + } + + entry = X509_NAME_get_entry(sname, i); + str = X509_NAME_ENTRY_get_data(entry); + + ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0, + "SSL commonName: \"%*s\"", + ASN1_STRING_length(str), ASN1_STRING_data(str)); + + if (ngx_ssl_check_name(name, str) == NGX_OK) { + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, + "SSL commonName: match"); + goto found; + } + } + + ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, + "SSL commonName: no match"); + } +#endif + +failed: + + X509_free(cert); + return NGX_ERROR; + +found: + + X509_free(cert); + return NGX_OK; +} + + +#if OPENSSL_VERSION_NUMBER < 0x10002001L + +static ngx_int_t +ngx_ssl_check_name(ngx_str_t *name, ASN1_STRING *pattern) +{ + u_char *s, *p, *end; + size_t slen, plen; + + s = name->data; + slen = name->len; + + p = ASN1_STRING_data(pattern); + plen = ASN1_STRING_length(pattern); + + if (slen == plen && ngx_strncasecmp(s, p, plen) == 0) { + return NGX_OK; + } + + if (plen > 2 && p[0] == '*' && p[1] == '.') { + plen -= 1; + p += 1; + + end = s + slen; + s = ngx_strlchr(s, end, '.'); + slen = end - s; + + if (plen == slen && ngx_strncasecmp(s, p, plen) == 0) { + return NGX_OK; + } + } + + return NGX_ERROR; +} + +#endif + + +ngx_int_t ngx_ssl_get_protocol(ngx_connection_t *c, ngx_pool_t *pool, ngx_str_t *s) { s->data = (u_char *) SSL_get_version(c->ssl->connection); diff --git a/src/event/ngx_event_openssl.h b/src/event/ngx_event_openssl.h --- a/src/event/ngx_event_openssl.h +++ b/src/event/ngx_event_openssl.h @@ -150,6 +150,8 @@ ngx_int_t ngx_ssl_set_session(ngx_connec || n == X509_V_ERR_CERT_UNTRUSTED \ || n == X509_V_ERR_UNABLE_TO_VERIFY_LEAF_SIGNATURE) +ngx_int_t ngx_ssl_check_host(ngx_connection_t *c, ngx_str_t *name); + ngx_int_t ngx_ssl_get_protocol(ngx_connection_t *c, ngx_pool_t *pool, ngx_str_t *s); diff --git a/src/http/modules/ngx_http_proxy_module.c b/src/http/modules/ngx_http_proxy_module.c --- a/src/http/modules/ngx_http_proxy_module.c +++ b/src/http/modules/ngx_http_proxy_module.c @@ -81,6 +81,9 @@ typedef struct { ngx_uint_t ssl; ngx_uint_t ssl_protocols; ngx_str_t ssl_ciphers; + ngx_uint_t ssl_verify_depth; + ngx_str_t ssl_trusted_certificate; + ngx_str_t ssl_crl; #endif } ngx_http_proxy_loc_conf_t; @@ -567,6 +570,34 @@ static ngx_command_t ngx_http_proxy_com offsetof(ngx_http_proxy_loc_conf_t, upstream.ssl_server_name), NULL }, + { ngx_string("proxy_ssl_verify"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_proxy_loc_conf_t, upstream.ssl_verify), + NULL }, + + { ngx_string("proxy_ssl_verify_depth"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, + ngx_conf_set_num_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_proxy_loc_conf_t, ssl_verify_depth), + NULL }, + + { ngx_string("proxy_ssl_trusted_certificate"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, + ngx_conf_set_str_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_proxy_loc_conf_t, ssl_trusted_certificate), + NULL }, + + { ngx_string("proxy_ssl_crl"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, + ngx_conf_set_str_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_proxy_loc_conf_t, ssl_crl), + NULL }, + #endif ngx_null_command @@ -2418,6 +2449,8 @@ ngx_http_proxy_create_loc_conf(ngx_conf_ * conf->ssl = 0; * conf->ssl_protocols = 0; * conf->ssl_ciphers = { 0, NULL }; + * conf->ssl_trusted_certificate = { 0, NULL }; + * conf->ssl_crl = { 0, NULL }; */ conf->upstream.store = NGX_CONF_UNSET; @@ -2460,6 +2493,8 @@ ngx_http_proxy_create_loc_conf(ngx_conf_ #if (NGX_HTTP_SSL) conf->upstream.ssl_session_reuse = NGX_CONF_UNSET; conf->upstream.ssl_server_name = NGX_CONF_UNSET; + conf->upstream.ssl_verify = NGX_CONF_UNSET; + conf->ssl_verify_depth = NGX_CONF_UNSET_UINT; #endif /* "proxy_cyclic_temp_file" is disabled */ @@ -2749,6 +2784,13 @@ ngx_http_proxy_merge_loc_conf(ngx_conf_t ngx_conf_merge_value(conf->upstream.ssl_server_name, prev->upstream.ssl_server_name, 0); + ngx_conf_merge_value(conf->upstream.ssl_verify, + prev->upstream.ssl_verify, 0); + ngx_conf_merge_uint_value(conf->ssl_verify_depth, + prev->ssl_verify_depth, 1); + ngx_conf_merge_str_value(conf->ssl_trusted_certificate, + prev->ssl_trusted_certificate, ""); + ngx_conf_merge_str_value(conf->ssl_crl, prev->ssl_crl, ""); if (conf->ssl && ngx_http_proxy_set_ssl(cf, conf) != NGX_OK) { return NGX_CONF_ERROR; @@ -3818,6 +3860,26 @@ ngx_http_proxy_set_ssl(ngx_conf_t *cf, n return NGX_ERROR; } + if (plcf->upstream.ssl_verify) { + if (plcf->ssl_trusted_certificate.len == 0) { + ngx_log_error(NGX_LOG_EMERG, cf->log, 0, + "no proxy_ssl_trusted_certificate for proxy_ssl_verify"); + return NGX_ERROR; + } + + if (ngx_ssl_trusted_certificate(cf, plcf->upstream.ssl, + &plcf->ssl_trusted_certificate, + plcf->ssl_verify_depth) + != NGX_OK) + { + return NGX_ERROR; + } + + if (ngx_ssl_crl(cf, plcf->upstream.ssl, &plcf->ssl_crl) != NGX_OK) { + return NGX_ERROR; + } + } + return NGX_OK; } From mdounin at mdounin.ru Fri Apr 18 16:33:42 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 18 Apr 2014 16:33:42 +0000 Subject: [nginx] Upstream: uwsgi_ssl_name, uwsgi_ssl_verify, and so on. Message-ID: details: http://hg.nginx.org/nginx/rev/3cb79707516a branches: changeset: 5662:3cb79707516a user: Maxim Dounin date: Fri Apr 18 20:13:32 2014 +0400 description: Upstream: uwsgi_ssl_name, uwsgi_ssl_verify, and so on. Just a merge of proxy_ssl_name, proxy_ssl_verify commits into uwsgi module, code is identical. diffstat: src/http/modules/ngx_http_uwsgi_module.c | 85 ++++++++++++++++++++++++++++++++ 1 files changed, 85 insertions(+), 0 deletions(-) diffs (139 lines): diff --git a/src/http/modules/ngx_http_uwsgi_module.c b/src/http/modules/ngx_http_uwsgi_module.c --- a/src/http/modules/ngx_http_uwsgi_module.c +++ b/src/http/modules/ngx_http_uwsgi_module.c @@ -39,6 +39,9 @@ typedef struct { ngx_uint_t ssl; ngx_uint_t ssl_protocols; ngx_str_t ssl_ciphers; + ngx_uint_t ssl_verify_depth; + ngx_str_t ssl_trusted_certificate; + ngx_str_t ssl_crl; #endif } ngx_http_uwsgi_loc_conf_t; @@ -409,6 +412,48 @@ static ngx_command_t ngx_http_uwsgi_comm offsetof(ngx_http_uwsgi_loc_conf_t, ssl_ciphers), NULL }, + { ngx_string("uwsgi_ssl_name"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, + ngx_http_set_complex_value_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_uwsgi_loc_conf_t, upstream.ssl_name), + NULL }, + + { ngx_string("uwsgi_ssl_server_name"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_uwsgi_loc_conf_t, upstream.ssl_server_name), + NULL }, + + { ngx_string("uwsgi_ssl_verify"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_uwsgi_loc_conf_t, upstream.ssl_verify), + NULL }, + + { ngx_string("uwsgi_ssl_verify_depth"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, + ngx_conf_set_num_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_uwsgi_loc_conf_t, ssl_verify_depth), + NULL }, + + { ngx_string("uwsgi_ssl_trusted_certificate"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, + ngx_conf_set_str_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_uwsgi_loc_conf_t, ssl_trusted_certificate), + NULL }, + + { ngx_string("uwsgi_ssl_crl"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, + ngx_conf_set_str_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_uwsgi_loc_conf_t, ssl_crl), + NULL }, + #endif ngx_null_command @@ -1243,8 +1288,12 @@ ngx_http_uwsgi_create_loc_conf(ngx_conf_ conf->upstream.pass_headers = NGX_CONF_UNSET_PTR; conf->upstream.intercept_errors = NGX_CONF_UNSET; + #if (NGX_HTTP_SSL) conf->upstream.ssl_session_reuse = NGX_CONF_UNSET; + conf->upstream.ssl_server_name = NGX_CONF_UNSET; + conf->upstream.ssl_verify = NGX_CONF_UNSET; + conf->ssl_verify_depth = NGX_CONF_UNSET_UINT; #endif /* "uwsgi_cyclic_temp_file" is disabled */ @@ -1494,6 +1543,7 @@ ngx_http_uwsgi_merge_loc_conf(ngx_conf_t prev->upstream.intercept_errors, 0); #if (NGX_HTTP_SSL) + ngx_conf_merge_value(conf->upstream.ssl_session_reuse, prev->upstream.ssl_session_reuse, 1); @@ -1505,6 +1555,20 @@ ngx_http_uwsgi_merge_loc_conf(ngx_conf_t ngx_conf_merge_str_value(conf->ssl_ciphers, prev->ssl_ciphers, "DEFAULT"); + if (conf->upstream.ssl_name == NULL) { + conf->upstream.ssl_name = prev->upstream.ssl_name; + } + + ngx_conf_merge_value(conf->upstream.ssl_server_name, + prev->upstream.ssl_server_name, 0); + ngx_conf_merge_value(conf->upstream.ssl_verify, + prev->upstream.ssl_verify, 0); + ngx_conf_merge_uint_value(conf->ssl_verify_depth, + prev->ssl_verify_depth, 1); + ngx_conf_merge_str_value(conf->ssl_trusted_certificate, + prev->ssl_trusted_certificate, ""); + ngx_conf_merge_str_value(conf->ssl_crl, prev->ssl_crl, ""); + if (conf->ssl && ngx_http_uwsgi_set_ssl(cf, conf) != NGX_OK) { return NGX_CONF_ERROR; } @@ -1512,6 +1576,7 @@ ngx_http_uwsgi_merge_loc_conf(ngx_conf_t if (conf->upstream.ssl == NULL) { conf->upstream.ssl = prev->upstream.ssl; } + #endif ngx_conf_merge_str_value(conf->uwsgi_string, prev->uwsgi_string, ""); @@ -2030,6 +2095,26 @@ ngx_http_uwsgi_set_ssl(ngx_conf_t *cf, n return NGX_ERROR; } + if (uwcf->upstream.ssl_verify) { + if (uwcf->ssl_trusted_certificate.len == 0) { + ngx_log_error(NGX_LOG_EMERG, cf->log, 0, + "no uwsgi_ssl_trusted_certificate for uwsgi_ssl_verify"); + return NGX_ERROR; + } + + if (ngx_ssl_trusted_certificate(cf, uwcf->upstream.ssl, + &uwcf->ssl_trusted_certificate, + uwcf->ssl_verify_depth) + != NGX_OK) + { + return NGX_ERROR; + } + + if (ngx_ssl_crl(cf, uwcf->upstream.ssl, &uwcf->ssl_crl) != NGX_OK) { + return NGX_ERROR; + } + } + return NGX_OK; } From mdounin at mdounin.ru Fri Apr 18 16:52:20 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 18 Apr 2014 20:52:20 +0400 Subject: [PATCH] SSL: add "{proxy, uwsgi}_ssl_verify" and supporting directives In-Reply-To: <20140212162908.GK38830@mdounin.ru> References: <20140206160955.GH1835@mdounin.ru> <20140207000319.GL1835@mdounin.ru> <20140207105818.GQ1835@mdounin.ru> <20140212162908.GK38830@mdounin.ru> Message-ID: <20140418165220.GT34696@mdounin.ru> Hello! On Wed, Feb 12, 2014 at 08:29:08PM +0400, Maxim Dounin wrote: > On Tue, Feb 11, 2014 at 01:16:41PM -0800, Piotr Sikora wrote: [...] > > > My original suggestion is as follows: > > > > > > proxy_ssl_name > > > > > > default: $proxy_host > > > complex value, controls a name used in SNI (if > > > enabled) > > > > > > proxy_ssl_verify on|off > > > > > > default: off > > > flag, controls if remote certificate verification is enabled > > > > > > proxy_ssl_verify_name on|off > > > > > > default: on > > > flag, controls if remote certificate verification needs to > > > check peer's name; must be explicitly switched off > > > if certificate verification is switched on, but > > > the name can't be checked due to too old OpenSSL > > > > Got it. > > Just a quick note: > > We've discussed this with Igor, and he thinks that peer's name > should be always checked, without an ability to check switch the > check off selectively. Mostly to simplify user experience. This > implies that we either need our own peer's name check code, or > verification won't work at all if OpenSSL is too old. Another quick note: I've committed backend SSL certificate verification code done which mostly matches the above description: http://hg.nginx.org/nginx/rev/7022564a9e0e http://hg.nginx.org/nginx/rev/060c2e692b96 -- Maxim Dounin http://nginx.org/ From ru at nginx.com Mon Apr 21 12:13:13 2014 From: ru at nginx.com (Ruslan Ermilov) Date: Mon, 21 Apr 2014 12:13:13 +0000 Subject: [nginx] Documented the switch from java XSLScript to xslscript.pl. Message-ID: details: http://hg.nginx.org/nginx/rev/001beb963d86 branches: changeset: 5663:001beb963d86 user: Ruslan Ermilov date: Mon Apr 21 13:48:41 2014 +0400 description: Documented the switch from java XSLScript to xslscript.pl. diffstat: misc/README | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r 3cb79707516a -r 001beb963d86 misc/README --- a/misc/README Fri Apr 18 20:13:32 2014 +0400 +++ b/misc/README Mon Apr 21 13:48:41 2014 +0400 @@ -3,7 +3,7 @@ make -f misc/GNUmakefile release the required tools: *) xsltproc to build CHANGES, -*) java and XSLScript ( http://pault.com/pault/XSLScript/ ) to build XSLTs +*) xslscript.pl ( http://hg.nginx.org/xslscript ) to build XSLTs from XSLScript sources. From mdounin at mdounin.ru Mon Apr 21 12:26:47 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 21 Apr 2014 16:26:47 +0400 Subject: [nginx] Adding Support for Weak ETags In-Reply-To: References: Message-ID: <20140421122647.GB34696@mdounin.ru> Hello! On Thu, Apr 17, 2014 at 05:39:40PM -0700, Aaron Peschel wrote: > Hello, > > I am interested in getting support for Weak ETags into the mainline. > There was some discussion previously in here previously that developed > a quick patch to add support. What additional functionality would be > required and what steps should be followed to get weak etag > functionality added to nginx? I am willing to do the work, I just need > some help with heading in the right direction. I had a quick draft patch sitting in my patchqueue since previous discussion (see [1]) to downgrade strict etags to weak ones. It needs more work though, as I'm not yet happy with the code. I hope I'll be able to find some time and finish it in 1.7.x. [1] http://mailman.nginx.org/pipermail/nginx-devel/2013-November/004523.html -- Maxim Dounin http://nginx.org/ From vbart at nginx.com Mon Apr 21 15:23:45 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Mon, 21 Apr 2014 15:23:45 +0000 Subject: [nginx] SPDY: Stream-ID restrictions according to specification. Message-ID: details: http://hg.nginx.org/nginx/rev/de93bbe2d8cd branches: changeset: 5664:de93bbe2d8cd user: Valentin Bartenev date: Mon Apr 21 18:59:53 2014 +0400 description: SPDY: Stream-ID restrictions according to specification. diffstat: src/http/ngx_http_spdy.c | 23 +++++++++++++++++++++-- 1 files changed, 21 insertions(+), 2 deletions(-) diffs (40 lines): diff -r 001beb963d86 -r de93bbe2d8cd src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Mon Apr 21 13:48:41 2014 +0400 +++ b/src/http/ngx_http_spdy.c Mon Apr 21 18:59:53 2014 +0400 @@ -937,6 +937,27 @@ ngx_http_spdy_state_syn_stream(ngx_http_ ngx_log_debug2(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, "spdy SYN_STREAM frame sid:%ui prio:%ui", sid, prio); + if (sid % 2 == 0 || sid <= sc->last_sid) { + ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, + "client sent SYN_STREAM frame " + "with invalid Stream-ID %ui", sid); + + stream = ngx_http_spdy_get_stream_by_id(sc, sid); + + if (stream) { + if (ngx_http_spdy_terminate_stream(sc, stream, + NGX_SPDY_PROTOCOL_ERROR) + != NGX_OK) + { + return ngx_http_spdy_state_internal_error(sc); + } + } + + return ngx_http_spdy_state_protocol_error(sc); + } + + sc->last_sid = sid; + sscf = ngx_http_get_module_srv_conf(sc->http_connection->conf_ctx, ngx_http_spdy_module); @@ -968,8 +989,6 @@ ngx_http_spdy_state_syn_stream(ngx_http_ sc->stream = stream; - sc->last_sid = sid; - return ngx_http_spdy_state_headers(sc, pos, end); } From vbart at nginx.com Mon Apr 21 15:23:47 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Mon, 21 Apr 2014 15:23:47 +0000 Subject: [nginx] SPDY: avoid sending RST_STREAM on WINDOW_UPDATE with unk... Message-ID: details: http://hg.nginx.org/nginx/rev/46d0795a846c branches: changeset: 5665:46d0795a846c user: Valentin Bartenev date: Mon Apr 21 19:21:17 2014 +0400 description: SPDY: avoid sending RST_STREAM on WINDOW_UPDATE with unknown SID. There's a race condition between closing a stream by one endpoint and sending a WINDOW_UPDATE frame by another. So it would be better to just skip such frames for unknown streams, like is already done for the DATA frames. diffstat: src/http/ngx_http_spdy.c | 12 ++---------- 1 files changed, 2 insertions(+), 10 deletions(-) diffs (22 lines): diff -r de93bbe2d8cd -r 46d0795a846c src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Mon Apr 21 18:59:53 2014 +0400 +++ b/src/http/ngx_http_spdy.c Mon Apr 21 19:21:17 2014 +0400 @@ -1341,16 +1341,8 @@ ngx_http_spdy_state_window_update(ngx_ht stream = ngx_http_spdy_get_stream_by_id(sc, sid); if (stream == NULL) { - ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, - "client sent WINDOW_UPDATE frame " - "for unknown stream %ui", sid); - - if (ngx_http_spdy_send_rst_stream(sc, sid, NGX_SPDY_INVALID_STREAM, - NGX_SPDY_LOWEST_PRIORITY) - == NGX_ERROR) - { - return ngx_http_spdy_state_internal_error(sc); - } + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, + "unknown spdy stream"); return ngx_http_spdy_state_complete(sc, pos, end); } From piotr at cloudflare.com Tue Apr 22 11:59:21 2014 From: piotr at cloudflare.com (Piotr Sikora) Date: Tue, 22 Apr 2014 04:59:21 -0700 Subject: [PATCH] SSL: support automatic selection of ECDH temporary key parameters In-Reply-To: <20140416100013.GY34696@mdounin.ru> References: <20140401170037.GZ34696@mdounin.ru> <20140416100013.GY34696@mdounin.ru> Message-ID: Hey Maxim, I've added the requested fallback to the "old code". It's not the perfect solution because OpenSSL returns the same response code for "SSL_CTX_set1_curves_list() not supported" and "invalid curves" cases, but it works. I verified that it behaves correctly when compiled against OpenSSL-1.0.2 but linked against OpenSSL-1.0.1, etc. I've also changed error messages to match the style of other SSL errors, but to be honest, it now looks strange, since the "new code" errors out with: SSL_CTX_set1_curves_list("XXX") failed whereas the "old code" errors out with: Unknown curve name "XXX" so we have 2 different error messages for the same error... Please let me know if I misunderstood your previous comment regarding those messages. Best regards, Piotr Sikora # HG changeset patch # User Piotr Sikora # Date 1398167365 25200 # Tue Apr 22 04:49:25 2014 -0700 # Node ID 27a8f0aacdff33b0dd6684815c1403cfd91ca895 # Parent 46d0795a846cc0ee5db8df68003d79ee918eed88 SSL: support automatic selection of ECDH temporary key parameters. When compiled against OpenSSL-1.0.2+, the colon separated list of supported curves can be provided using either curve NIDs: ssl_ecdh_curve secp521r1:secp384r1:prime256v1; or names: ssl_ecdh_curve P-521:P-384:P-256; Signed-off-by: Piotr Sikora diff -r 46d0795a846c -r 27a8f0aacdff src/event/ngx_event_openssl.c --- a/src/event/ngx_event_openssl.c Mon Apr 21 19:21:17 2014 +0400 +++ b/src/event/ngx_event_openssl.c Tue Apr 22 04:49:25 2014 -0700 @@ -685,6 +685,30 @@ ngx_ssl_ecdh_curve(ngx_conf_t *cf, ngx_s { #if OPENSSL_VERSION_NUMBER >= 0x0090800fL #ifndef OPENSSL_NO_ECDH +#ifdef SSL_CTRL_SET_ECDH_AUTO + + if (SSL_CTX_set1_curves_list(ssl->ctx, "prime256v1") == 1) { + + if (SSL_CTX_set1_curves_list(ssl->ctx, name->data) == 0) { + ngx_ssl_error(NGX_LOG_EMERG, ssl->log, 0, + "SSL_CTX_set1_curves_list(\"%s\") failed", + name->data); + return NGX_ERROR; + } + + SSL_CTX_set_options(ssl->ctx, SSL_OP_SINGLE_ECDH_USE); + + if (SSL_CTX_set_ecdh_auto(ssl->ctx, 1) == 0) { + ngx_ssl_error(NGX_LOG_EMERG, ssl->log, 0, + "SSL_CTX_set_ecdh_auto() failed"); + return NGX_ERROR; + } + + return NGX_OK; + } + +#endif + { int nid; EC_KEY *ecdh; @@ -714,6 +738,7 @@ ngx_ssl_ecdh_curve(ngx_conf_t *cf, ngx_s SSL_CTX_set_tmp_ecdh(ssl->ctx, ecdh); EC_KEY_free(ecdh); + } #endif #endif From piotr at cloudflare.com Tue Apr 22 12:00:51 2014 From: piotr at cloudflare.com (Piotr Sikora) Date: Tue, 22 Apr 2014 05:00:51 -0700 Subject: [PATCH] SSL: add "{proxy, uwsgi}_ssl_verify" and supporting directives In-Reply-To: <20140418165220.GT34696@mdounin.ru> References: <20140206160955.GH1835@mdounin.ru> <20140207000319.GL1835@mdounin.ru> <20140207105818.GQ1835@mdounin.ru> <20140212162908.GK38830@mdounin.ru> <20140418165220.GT34696@mdounin.ru> Message-ID: Hey Maxim, > Another quick note: > > I've committed backend SSL certificate verification code done > which mostly matches the above description: > > http://hg.nginx.org/nginx/rev/7022564a9e0e > http://hg.nginx.org/nginx/rev/060c2e692b96 Thanks... and sorry for the delay with that code. Best regards, Piotr Sikora From mdounin at mdounin.ru Tue Apr 22 12:11:57 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 22 Apr 2014 12:11:57 +0000 Subject: [nginx] SSL: added explicit check for ngx_strlchr() result. Message-ID: details: http://hg.nginx.org/nginx/rev/a77c0839c993 branches: changeset: 5666:a77c0839c993 user: Maxim Dounin date: Tue Apr 22 14:02:45 2014 +0400 description: SSL: added explicit check for ngx_strlchr() result. diffstat: src/event/ngx_event_openssl.c | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diffs (15 lines): diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c --- a/src/event/ngx_event_openssl.c +++ b/src/event/ngx_event_openssl.c @@ -2637,6 +2637,11 @@ ngx_ssl_check_name(ngx_str_t *name, ASN1 end = s + slen; s = ngx_strlchr(s, end, '.'); + + if (s == NULL) { + return NGX_ERROR; + } + slen = end - s; if (plen == slen && ngx_strncasecmp(s, p, plen) == 0) { From ru at nginx.com Tue Apr 22 14:58:35 2014 From: ru at nginx.com (Ruslan Ermilov) Date: Tue, 22 Apr 2014 14:58:35 +0000 Subject: [nginx] Upstream: for ssl name, non-aligned memory allocation is... Message-ID: details: http://hg.nginx.org/nginx/rev/16405e02e612 branches: changeset: 5667:16405e02e612 user: Ruslan Ermilov date: Tue Apr 22 18:56:49 2014 +0400 description: Upstream: for ssl name, non-aligned memory allocation is enough. diffstat: src/http/ngx_http_upstream.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r a77c0839c993 -r 16405e02e612 src/http/ngx_http_upstream.c --- a/src/http/ngx_http_upstream.c Tue Apr 22 14:02:45 2014 +0400 +++ b/src/http/ngx_http_upstream.c Tue Apr 22 18:56:49 2014 +0400 @@ -1511,7 +1511,7 @@ ngx_http_upstream_ssl_name(ngx_http_requ * hence we explicitly null-terminate name here */ - p = ngx_palloc(r->pool, name.len + 1); + p = ngx_pnalloc(r->pool, name.len + 1); if (p == NULL) { return NGX_ERROR; } From maxim at nginx.com Wed Apr 23 16:01:07 2014 From: maxim at nginx.com (Maxim Konovalov) Date: Wed, 23 Apr 2014 16:01:07 +0000 Subject: [nginx] Missed comma fixed. A couple of Latin symbols plugged into Message-ID: details: http://hg.nginx.org/nginx/rev/64958ea92fd2 branches: changeset: 5668:64958ea92fd2 user: Maxim Konovalov date: Wed Apr 23 16:00:59 2014 +0000 description: Missed comma fixed. A couple of Latin symbols plugged into Russian text changed to Russian counterparts. diffstat: docs/xml/nginx/changes.xml | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diffs (30 lines): diff -r 16405e02e612 -r 64958ea92fd2 docs/xml/nginx/changes.xml --- a/docs/xml/nginx/changes.xml Tue Apr 22 18:56:49 2014 +0400 +++ b/docs/xml/nginx/changes.xml Wed Apr 23 16:00:59 2014 +0000 @@ -673,7 +673,7 @@ in error handling in ngx_http_gunzip_fil -???????????? ?????????? ???????????????? +???????????? ?????????? ????????????????, ???????? ?????????????????????????? ???????????? ngx_http_spdy_module ?? ?????????????????? auth_request. @@ -2293,7 +2293,7 @@ the "single" parameter of the "keepalive ???????????? SSL ???????????? ?????????????????? -?? ?????? ?????????? ?????? ?????????????????????????? OpenSSL c?????????? 1.0.0. +?? ?????? ?????????? ?????? ?????????????????????????? OpenSSL ???????????? 1.0.0. SSL compression is now disabled when using all versions of OpenSSL, @@ -21030,7 +21030,7 @@ the %3F symbol in the URI was considered -?????????????????? unix domain ??o?????????? ?? ???????????? ngx_http_proxy_module. +?????????????????? unix domain ?????????????? ?? ???????????? ngx_http_proxy_module. the unix domain sockets support in the ngx_http_proxy_module. From mdounin at mdounin.ru Wed Apr 23 16:32:08 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 23 Apr 2014 16:32:08 +0000 Subject: [nginx] SSL: explicit handling of empty names. Message-ID: details: http://hg.nginx.org/nginx/rev/cac82b9b3499 branches: changeset: 5669:cac82b9b3499 user: Maxim Dounin date: Wed Apr 23 20:31:31 2014 +0400 description: SSL: explicit handling of empty names. X509_check_host() can't handle non null-terminated names with zero length, so make sure to fail before calling it. diffstat: src/event/ngx_event_openssl.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diffs (14 lines): diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c --- a/src/event/ngx_event_openssl.c +++ b/src/event/ngx_event_openssl.c @@ -2504,6 +2504,10 @@ ngx_ssl_check_host(ngx_connection_t *c, /* X509_check_host() is only available in OpenSSL 1.0.2+ */ + if (name->len == 0) { + goto failed; + } + if (X509_check_host(cert, name->data, name->len, 0) != 1) { ngx_log_debug0(NGX_LOG_DEBUG_EVENT, c->log, 0, "X509_check_host(): no match"); From mdounin at mdounin.ru Wed Apr 23 17:16:23 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 23 Apr 2014 17:16:23 +0000 Subject: [nginx] Missed comma fixed. A couple of Latin symbols plugged into Message-ID: details: http://hg.nginx.org/nginx/rev/ec4ca22bfb86 branches: stable-1.6 changeset: 5670:ec4ca22bfb86 user: Maxim Konovalov date: Wed Apr 23 16:00:59 2014 +0000 description: Missed comma fixed. A couple of Latin symbols plugged into Russian text changed to Russian counterparts. diffstat: docs/xml/nginx/changes.xml | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diffs (30 lines): diff --git a/docs/xml/nginx/changes.xml b/docs/xml/nginx/changes.xml --- a/docs/xml/nginx/changes.xml +++ b/docs/xml/nginx/changes.xml @@ -673,7 +673,7 @@ in error handling in ngx_http_gunzip_fil -?????? ????? ???????? +?????? ????? ????????, ???? ????????????? ?????? ngx_http_spdy_module ? ????????? auth_request. @@ -2293,7 +2293,7 @@ the "single" parameter of the "keepalive ?????? SSL ?????? ????????? -? ??? ????? ??? ????????????? OpenSSL c????? 1.0.0. +? ??? ????? ??? ????????????? OpenSSL ?????? 1.0.0. SSL compression is now disabled when using all versions of OpenSSL, @@ -21030,7 +21030,7 @@ the %3F symbol in the URI was considered -????????? unix domain ?o????? ? ?????? ngx_http_proxy_module. +????????? unix domain ??????? ? ?????? ngx_http_proxy_module. the unix domain sockets support in the ngx_http_proxy_module. From piotr at cloudflare.com Wed Apr 23 21:32:18 2014 From: piotr at cloudflare.com (Piotr Sikora) Date: Wed, 23 Apr 2014 14:32:18 -0700 Subject: [PATCH] Configure: call "make distclean" for zlib only if Makefile exists Message-ID: # HG changeset patch # User Piotr Sikora # Date 1398288630 25200 # Wed Apr 23 14:30:30 2014 -0700 # Node ID c4c006176ad05e498bdbddb8db41ce33733afedc # Parent cac82b9b3499922de352627ccd7f10f6dac66586 Configure: call "make distclean" for zlib only if Makefile exists. This change allows to build nginx against git checkout of zlib. Signed-off-by: Piotr Sikora diff -r cac82b9b3499 -r c4c006176ad0 auto/lib/zlib/make --- a/auto/lib/zlib/make Wed Apr 23 20:31:31 2014 +0400 +++ b/auto/lib/zlib/make Wed Apr 23 14:30:30 2014 -0700 @@ -52,7 +52,7 @@ END $ZLIB/libz.a: $NGX_MAKEFILE cd $ZLIB \\ - && \$(MAKE) distclean \\ + && if [ -f Makefile ]; then \$(MAKE) distclean; fi \\ && \$(MAKE) -f win32/Makefile.gcc \\ CFLAGS="$ZLIB_OPT" CC="\$(CC)" \\ libz.a @@ -75,7 +75,7 @@ END $ZLIB/libz.a: $NGX_MAKEFILE cd $ZLIB \\ - && \$(MAKE) distclean \\ + && if [ -f Makefile ]; then \$(MAKE) distclean; fi \\ && cp contrib/asm586/match.S . \\ && CFLAGS="$ZLIB_OPT -DASMV" CC="\$(CC)" \\ ./configure \\ @@ -92,7 +92,7 @@ END $ZLIB/libz.a: $NGX_MAKEFILE cd $ZLIB \\ - && \$(MAKE) distclean \\ + && if [ -f Makefile ]; then \$(MAKE) distclean; fi \\ && cp contrib/asm686/match.S . \\ && CFLAGS="$ZLIB_OPT -DASMV" CC="\$(CC)" \\ ./configure \\ @@ -125,7 +125,7 @@ if [ $done = NO ]; then $ZLIB/libz.a: $NGX_MAKEFILE cd $ZLIB \\ - && \$(MAKE) distclean \\ + && if [ -f Makefile ]; then \$(MAKE) distclean; fi \\ && CFLAGS="$ZLIB_OPT" CC="\$(CC)" \\ ./configure \\ && \$(MAKE) libz.a From piotr at cloudflare.com Thu Apr 24 10:16:15 2014 From: piotr at cloudflare.com (Piotr Sikora) Date: Thu, 24 Apr 2014 03:16:15 -0700 Subject: [PATCH] Configure: call "make distclean" for zlib only if Makefile exists In-Reply-To: References: Message-ID: Hey, > Configure: call "make distclean" for zlib only if Makefile exists. > > This change allows to build nginx against git checkout of zlib. Actually, it looks that only our amd64-optimized version [1] doesn't ship with Makefile, the original repository [2] has a minimal one with "distclean" target, so feel free to ignore this patch (although, I still think it's a good practice to check if the Makefile exists before calling make). [1] https://github.com/cloudflare/zlib/commits/gcc.amd64 [2] https://github.com/madler/zlib Best regards, Piotr Sikora From mdounin at mdounin.ru Thu Apr 24 13:01:57 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 24 Apr 2014 13:01:57 +0000 Subject: [nginx] nginx-1.6.0-RELEASE Message-ID: details: http://hg.nginx.org/nginx/rev/daa5384fd526 branches: stable-1.6 changeset: 5671:daa5384fd526 user: Maxim Dounin date: Thu Apr 24 16:52:24 2014 +0400 description: nginx-1.6.0-RELEASE diffstat: docs/xml/nginx/changes.xml | 14 ++++++++++++++ 1 files changed, 14 insertions(+), 0 deletions(-) diffs (24 lines): diff --git a/docs/xml/nginx/changes.xml b/docs/xml/nginx/changes.xml --- a/docs/xml/nginx/changes.xml +++ b/docs/xml/nginx/changes.xml @@ -5,6 +5,20 @@ + + + + +?????????? ????? 1.6.x. + + +1.6.x stable branch. + + + + + + From mdounin at mdounin.ru Thu Apr 24 13:01:58 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 24 Apr 2014 13:01:58 +0000 Subject: [nginx] release-1.6.0 tag Message-ID: details: http://hg.nginx.org/nginx/rev/622887a470d3 branches: stable-1.6 changeset: 5672:622887a470d3 user: Maxim Dounin date: Thu Apr 24 16:52:24 2014 +0400 description: release-1.6.0 tag diffstat: .hgtags | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (8 lines): diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -369,3 +369,4 @@ b798fc020e3a84ef68e6c9f47865a319c826d33c f995a10d4c7e9a817157a6ce7b753297ad32897e release-1.5.11 97b47d95e4449cbde976657cf8cbbc118351ffe0 release-1.5.12 fd722b890eabc600394349730a093f50dac31639 release-1.5.13 +daa5384fd526a9c18fff4f5135646743628f6bc7 release-1.6.0 From mdounin at mdounin.ru Thu Apr 24 13:01:59 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 24 Apr 2014 13:01:59 +0000 Subject: [nginx] nginx-1.7.0-RELEASE Message-ID: details: http://hg.nginx.org/nginx/rev/d161d68df8be branches: changeset: 5673:d161d68df8be user: Maxim Dounin date: Thu Apr 24 16:54:22 2014 +0400 description: nginx-1.7.0-RELEASE diffstat: docs/xml/nginx/changes.xml | 41 +++++++++++++++++++++++++++++++++++++++++ 1 files changed, 41 insertions(+), 0 deletions(-) diffs (51 lines): diff --git a/docs/xml/nginx/changes.xml b/docs/xml/nginx/changes.xml --- a/docs/xml/nginx/changes.xml +++ b/docs/xml/nginx/changes.xml @@ -5,6 +5,47 @@ + + + + +???????? SSL-???????????? ????????. + + +backend SSL certificate verification. + + + + + +????????? SNI ??? ?????? ? ????????? ?? SSL. + + +support for SNI while working with SSL backends. + + + + + +?????????? $ssl_server_name. + + +the $ssl_server_name variable. + + + + + +???????? if ????????? access_log. + + +the "if" parameter of the "access_log" directive. + + + + + + From mdounin at mdounin.ru Thu Apr 24 13:02:01 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 24 Apr 2014 13:02:01 +0000 Subject: [nginx] release-1.7.0 tag Message-ID: details: http://hg.nginx.org/nginx/rev/1b0c55d38d0b branches: changeset: 5674:1b0c55d38d0b user: Maxim Dounin date: Thu Apr 24 16:54:23 2014 +0400 description: release-1.7.0 tag diffstat: .hgtags | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (8 lines): diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -369,3 +369,4 @@ b798fc020e3a84ef68e6c9f47865a319c826d33c f995a10d4c7e9a817157a6ce7b753297ad32897e release-1.5.11 97b47d95e4449cbde976657cf8cbbc118351ffe0 release-1.5.12 fd722b890eabc600394349730a093f50dac31639 release-1.5.13 +d161d68df8be32e5cbf72b07db1a707714827803 release-1.7.0 From vbart at nginx.com Thu Apr 24 16:51:52 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Thu, 24 Apr 2014 16:51:52 +0000 Subject: [nginx] Version bump. Message-ID: details: http://hg.nginx.org/nginx/rev/1710bf72243e branches: changeset: 5675:1710bf72243e user: Valentin Bartenev date: Thu Apr 24 20:50:10 2014 +0400 description: Version bump. diffstat: src/core/nginx.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (14 lines): diff -r 1b0c55d38d0b -r 1710bf72243e src/core/nginx.h --- a/src/core/nginx.h Thu Apr 24 16:54:23 2014 +0400 +++ b/src/core/nginx.h Thu Apr 24 20:50:10 2014 +0400 @@ -9,8 +9,8 @@ #define _NGINX_H_INCLUDED_ -#define nginx_version 1007000 -#define NGINX_VERSION "1.7.0" +#define nginx_version 1007001 +#define NGINX_VERSION "1.7.1" #define NGINX_VER "nginx/" NGINX_VERSION #define NGINX_VAR "NGINX" From vbart at nginx.com Thu Apr 24 16:51:53 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Thu, 24 Apr 2014 16:51:53 +0000 Subject: [nginx] Proxy: fixed possible uninitialized memory access. Message-ID: details: http://hg.nginx.org/nginx/rev/fbfdf8017748 branches: changeset: 5676:fbfdf8017748 user: Valentin Bartenev date: Mon Nov 18 03:06:45 2013 +0400 description: Proxy: fixed possible uninitialized memory access. The ngx_http_proxy_rewrite_cookie() function expects the value of the "Set-Cookie" header to be null-terminated, and for headers obtained from proxied server it is usually true. Now the ngx_http_proxy_rewrite() function preserves the null character while rewriting headers. This fixes accessing memory outside of rewritten value if both the "proxy_cookie_path" and "proxy_cookie_domain" directives are used in the same location. diffstat: src/http/modules/ngx_http_proxy_module.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diffs (30 lines): diff -r 1710bf72243e -r fbfdf8017748 src/http/modules/ngx_http_proxy_module.c --- a/src/http/modules/ngx_http_proxy_module.c Thu Apr 24 20:50:10 2014 +0400 +++ b/src/http/modules/ngx_http_proxy_module.c Mon Nov 18 03:06:45 2013 +0400 @@ -2365,7 +2365,7 @@ ngx_http_proxy_rewrite(ngx_http_request_ if (replacement->len > len) { - data = ngx_pnalloc(r->pool, new_len); + data = ngx_pnalloc(r->pool, new_len + 1); if (data == NULL) { return NGX_ERROR; } @@ -2374,7 +2374,7 @@ ngx_http_proxy_rewrite(ngx_http_request_ p = ngx_copy(p, replacement->data, replacement->len); ngx_memcpy(p, h->value.data + prefix + len, - h->value.len - len - prefix); + h->value.len - len - prefix + 1); h->value.data = data; @@ -2383,7 +2383,7 @@ ngx_http_proxy_rewrite(ngx_http_request_ replacement->len); ngx_memmove(p, h->value.data + prefix + len, - h->value.len - len - prefix); + h->value.len - len - prefix + 1); } h->value.len = new_len; From quanah at zimbra.com Thu Apr 24 17:19:45 2014 From: quanah at zimbra.com (Quanah Gibson-Mount) Date: Thu, 24 Apr 2014 12:19:45 -0500 Subject: [PATCH 0 of 2] TRAC 531 - Add source port logging for HTTP(S) error logging and all proxied email client connections Message-ID: The following patches ensure that the source port is logged for all client connections. This is to resolve TRAC issue 531. Belgium is mandating that the source port be logged for all client connections for Carrier Grade NAT. This may soon extend to the entire European Union. From quanah at zimbra.com Thu Apr 24 17:19:47 2014 From: quanah at zimbra.com (Quanah Gibson-Mount) Date: Thu, 24 Apr 2014 12:19:47 -0500 Subject: [PATCH 2 of 2] MAIL: Always log the source port of the client In-Reply-To: References: Message-ID: <3c908c40acd15c8df020.1398359987@zre-ldap001.eng.zimbra.com> # HG changeset patch # User Quanah Gibson-Mount # Date 1398359069 18000 # Node ID 3c908c40acd15c8df020f95309b98d45f2b5e5de # Parent 4b7d2e503c06758330aabcc21ffbbab77f09568e MAIL: Always log the source port of the client For TRAC 531 diff -r 4b7d2e503c06 -r 3c908c40acd1 src/mail/ngx_mail.h --- a/src/mail/ngx_mail.h Thu Apr 24 11:39:17 2014 -0500 +++ b/src/mail/ngx_mail.h Thu Apr 24 12:04:29 2014 -0500 @@ -253,6 +253,7 @@ typedef struct { ngx_str_t *client; + ngx_uint_t client_port; ngx_mail_session_t *session; } ngx_mail_log_ctx_t; diff -r 4b7d2e503c06 -r 3c908c40acd1 src/mail/ngx_mail_handler.c --- a/src/mail/ngx_mail_handler.c Thu Apr 24 11:39:17 2014 -0500 +++ b/src/mail/ngx_mail_handler.c Thu Apr 24 12:04:29 2014 -0500 @@ -24,6 +24,7 @@ { ngx_uint_t i; ngx_mail_port_t *port; + ngx_uint_t remote_port=0; struct sockaddr *sa; struct sockaddr_in *sin; ngx_mail_log_ctx_t *ctx; @@ -127,8 +128,26 @@ c->data = s; s->connection = c; - ngx_log_error(NGX_LOG_INFO, c->log, 0, "*%uA client %V connected to %V", + switch (c->sockaddr->sa_family) { +#if (NGX_HAVE_INET6) + case AF_INET6: + sin6 = (struct sockaddr_in6 *) c->sockaddr; + remote_port = ntohs(sin6->sin6_port); + break; +#endif + default: + sin = (struct sockaddr_in *) c->sockaddr; + remote_port = ntohs(sin->sin_port); + break; + } + + if (remote_port && remote_port < 65536) { + ngx_log_error(NGX_LOG_INFO, c->log, 0, "*%uA client %V:%ui connected to %V", + c->number, &c->addr_text, remote_port, s->addr_text); + } else { + ngx_log_error(NGX_LOG_INFO, c->log, 0, "*%uA client %V connected to %V", c->number, &c->addr_text, s->addr_text); + } ctx = ngx_palloc(c->pool, sizeof(ngx_mail_log_ctx_t)); if (ctx == NULL) { @@ -137,6 +156,7 @@ } ctx->client = &c->addr_text; + ctx->client_port = remote_port; ctx->session = s; c->log->connection = c->number; @@ -750,7 +770,11 @@ ctx = log->data; - p = ngx_snprintf(buf, len, ", client: %V", ctx->client); + if (ctx->client_port && ctx->client_port < 65536) { + p = ngx_snprintf(buf, len, ", client: %V:%ui", ctx->client, ctx->client_port); + } else { + p = ngx_snprintf(buf, len, ", client: %V", ctx->client, ctx->client_port); + } len -= p - buf; buf = p; From quanah at zimbra.com Thu Apr 24 17:19:46 2014 From: quanah at zimbra.com (Quanah Gibson-Mount) Date: Thu, 24 Apr 2014 12:19:46 -0500 Subject: [PATCH 1 of 2] HTTP: Add client source port to any error that is logged In-Reply-To: References: Message-ID: <4b7d2e503c06758330aa.1398359986@zre-ldap001.eng.zimbra.com> # HG changeset patch # User Quanah Gibson-Mount # Date 1398357557 18000 # Node ID 4b7d2e503c06758330aabcc21ffbbab77f09568e # Parent 1b0c55d38d0b7ba69dcad79760a3fadc30696a9d HTTP: Add client source port to any error that is logged For TRAC ticket 531 diff -r 1b0c55d38d0b -r 4b7d2e503c06 src/http/ngx_http_request.c --- a/src/http/ngx_http_request.c Thu Apr 24 16:54:23 2014 +0400 +++ b/src/http/ngx_http_request.c Thu Apr 24 11:39:17 2014 -0500 @@ -3548,6 +3548,11 @@ u_char *p; ngx_http_request_t *r; ngx_http_log_ctx_t *ctx; + ngx_uint_t remote_port=0; + struct sockaddr_in *sin; +#if (NGX_HAVE_INET6) + struct sockaddr_in6 *sin6; +#endif if (log->action) { p = ngx_snprintf(buf, len, " while %s", log->action); @@ -3557,15 +3562,32 @@ ctx = log->data; - p = ngx_snprintf(buf, len, ", client: %V", &ctx->connection->addr_text); - len -= p - buf; - r = ctx->request; - if (r) { + switch (r->connection->sockaddr->sa_family) { + #if (NGX_HAVE_INET6) + case AF_INET6: + sin6 = (struct sockaddr_in6 *) r->connection->sockaddr; + remote_port = ntohs(sin6->sin6_port); + break; + #endif + + default: /* AF_INET */ + sin = (struct sockaddr_in *) r->connection->sockaddr; + remote_port = ntohs(sin->sin_port); + break; + } + + if (remote_port && remote_port < 65536) { + p = ngx_snprintf(buf, len, ", client: %V:%ui", &ctx->connection->addr_text,remote_port); + } else { + p = ngx_snprintf(buf, len, ", client: %V", &ctx->connection->addr_text); + } + len -= p - buf; + return r->log_handler(r, ctx->current_request, p, len); - } else { + p = ngx_snprintf(buf, len, ", client: %V", &ctx->connection->addr_text); p = ngx_snprintf(p, len, ", server: %V", &ctx->connection->listening->addr_text); } From mdounin at mdounin.ru Thu Apr 24 17:37:54 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 24 Apr 2014 21:37:54 +0400 Subject: [PATCH 1 of 2] HTTP: Add client source port to any error that is logged In-Reply-To: <4b7d2e503c06758330aa.1398359986@zre-ldap001.eng.zimbra.com> References: <4b7d2e503c06758330aa.1398359986@zre-ldap001.eng.zimbra.com> Message-ID: <20140424173753.GI34696@mdounin.ru> Hello! On Thu, Apr 24, 2014 at 12:19:46PM -0500, Quanah Gibson-Mount wrote: > # HG changeset patch > # User Quanah Gibson-Mount > # Date 1398357557 18000 > # Node ID 4b7d2e503c06758330aabcc21ffbbab77f09568e > # Parent 1b0c55d38d0b7ba69dcad79760a3fadc30696a9d > HTTP: Add client source port to any error that is logged > For TRAC ticket 531 I tend to say "No, thanks". If needed due to local regulations, $remote_port can be added to log_format. -- Maxim Dounin http://nginx.org/ From quanah at zimbra.com Thu Apr 24 17:41:43 2014 From: quanah at zimbra.com (Quanah Gibson-Mount) Date: Thu, 24 Apr 2014 10:41:43 -0700 Subject: [PATCH 1 of 2] HTTP: Add client source port to any error that is logged In-Reply-To: <20140424173753.GI34696@mdounin.ru> References: <4b7d2e503c06758330aa.1398359986@zre-ldap001.eng.zimbra.com> <20140424173753.GI34696@mdounin.ru> Message-ID: <861BE01750014CC67B7EF5A9@[192.168.1.15]> --On April 24, 2014 at 9:37:54 PM +0400 Maxim Dounin wrote: > I tend to say "No, thanks". > > If needed due to local regulations, $remote_port can be added to > log_format. $remote_port in the log format section only covers errors logged to the access log, it does not cover errors in the error log. The submitted patch handles the error log. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration From quanah at zimbra.com Thu Apr 24 17:44:37 2014 From: quanah at zimbra.com (Quanah Gibson-Mount) Date: Thu, 24 Apr 2014 10:44:37 -0700 Subject: [PATCH 1 of 2] HTTP: Add client source port to any error that is logged In-Reply-To: <861BE01750014CC67B7EF5A9@[192.168.1.15]> References: <4b7d2e503c06758330aa.1398359986@zre-ldap001.eng.zimbra.com> <20140424173753.GI34696@mdounin.ru> <861BE01750014CC67B7EF5A9@[192.168.1.15]> Message-ID: <96E01A7BADBD2CFD3D996021@[192.168.1.15]> --On April 24, 2014 at 10:41:43 AM -0700 Quanah Gibson-Mount wrote: > > > --On April 24, 2014 at 9:37:54 PM +0400 Maxim Dounin > wrote: >> I tend to say "No, thanks". >> >> If needed due to local regulations, $remote_port can be added to >> log_format. > > $remote_port in the log format section only covers errors logged to the > access log, accesses even.. :P --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration From mdounin at mdounin.ru Thu Apr 24 17:47:39 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 24 Apr 2014 21:47:39 +0400 Subject: [PATCH 2 of 2] MAIL: Always log the source port of the client In-Reply-To: <3c908c40acd15c8df020.1398359987@zre-ldap001.eng.zimbra.com> References: <3c908c40acd15c8df020.1398359987@zre-ldap001.eng.zimbra.com> Message-ID: <20140424174738.GJ34696@mdounin.ru> Hello! On Thu, Apr 24, 2014 at 12:19:47PM -0500, Quanah Gibson-Mount wrote: > # HG changeset patch > # User Quanah Gibson-Mount > # Date 1398359069 18000 > # Node ID 3c908c40acd15c8df020f95309b98d45f2b5e5de > # Parent 4b7d2e503c06758330aabcc21ffbbab77f09568e > MAIL: Always log the source port of the client > For TRAC 531 Much like http-related counterpart, this looks like a hack for me. We may consider adding port to the "client connected" messages (not sure), but I don't think we have to do anything beyond that. -- Maxim Dounin http://nginx.org/ From mdounin at mdounin.ru Thu Apr 24 17:56:48 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 24 Apr 2014 21:56:48 +0400 Subject: [PATCH 1 of 2] HTTP: Add client source port to any error that is logged In-Reply-To: <861BE01750014CC67B7EF5A9@[192.168.1.15]> References: <4b7d2e503c06758330aa.1398359986@zre-ldap001.eng.zimbra.com> <20140424173753.GI34696@mdounin.ru> <861BE01750014CC67B7EF5A9@[192.168.1.15]> Message-ID: <20140424175648.GK34696@mdounin.ru> Hello! On Thu, Apr 24, 2014 at 10:41:43AM -0700, Quanah Gibson-Mount wrote: > > > --On April 24, 2014 at 9:37:54 PM +0400 Maxim Dounin > wrote: > >I tend to say "No, thanks". > > > >If needed due to local regulations, $remote_port can be added to > >log_format. > > $remote_port in the log format section only covers errors logged to the > access log, it does not cover errors in the error log. The submitted patch > handles the error log. I understand the difference, thank you. The ticket in question only talked about error_log in context of mail module, where is no separate access logging to meet the alleged regulations. -- Maxim Dounin http://nginx.org/ From quanah at zimbra.com Thu Apr 24 18:06:29 2014 From: quanah at zimbra.com (Quanah Gibson-Mount) Date: Thu, 24 Apr 2014 11:06:29 -0700 Subject: [PATCH 1 of 2] HTTP: Add client source port to any error that is logged In-Reply-To: <20140424175648.GK34696@mdounin.ru> References: <4b7d2e503c06758330aa.1398359986@zre-ldap001.eng.zimbra.com> <20140424173753.GI34696@mdounin.ru> <861BE01750014CC67B7EF5A9@[192.168.1.15]> <20140424175648.GK34696@mdounin.ru> Message-ID: --On April 24, 2014 at 9:56:48 PM +0400 Maxim Dounin wrote: >> $remote_port in the log format section only covers errors logged to the >> access log, it does not cover errors in the error log. The submitted >> patch handles the error log. > > I understand the difference, thank you. > > The ticket in question only talked about error_log in context of > mail module, where is no separate access logging to meet the > alleged regulations. Yes, that is true, but why only implement a partial solution? With CGN, only logging the IP is fairly useless in all cases. To truly get useful information going forward, the IP + PORT of the client should logged in all cases. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration From mdounin at mdounin.ru Thu Apr 24 18:26:07 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 24 Apr 2014 22:26:07 +0400 Subject: [PATCH 1 of 2] HTTP: Add client source port to any error that is logged In-Reply-To: References: <4b7d2e503c06758330aa.1398359986@zre-ldap001.eng.zimbra.com> <20140424173753.GI34696@mdounin.ru> <861BE01750014CC67B7EF5A9@[192.168.1.15]> <20140424175648.GK34696@mdounin.ru> Message-ID: <20140424182607.GL34696@mdounin.ru> Hello! On Thu, Apr 24, 2014 at 11:06:29AM -0700, Quanah Gibson-Mount wrote: > > > --On April 24, 2014 at 9:56:48 PM +0400 Maxim Dounin > wrote: > > >>$remote_port in the log format section only covers errors logged to the > >>access log, it does not cover errors in the error log. The submitted > >>patch handles the error log. > > > >I understand the difference, thank you. > > > >The ticket in question only talked about error_log in context of > >mail module, where is no separate access logging to meet the > >alleged regulations. > > Yes, that is true, but why only implement a partial solution? With CGN, > only logging the IP is fairly useless in all cases. To truly get useful > information going forward, the IP + PORT of the client should logged in all > cases. Access log certainly can be configured to provide enough enformation to match any given error log message to a port if needed. There is no need to implement anything, solution is already here. And, by asking about "why implement a partical solution" you are overlooking the fact that proposed solution is partial as well - it doesn't change c->addr_text to ensure proper logging in all places (this would be a bad idea for other reasons, but it's another question), but rather tries to hack on the http error logging code to introduce remote port logging. This is far from being a complete solution. -- Maxim Dounin http://nginx.org/ From quanah at zimbra.com Thu Apr 24 23:37:31 2014 From: quanah at zimbra.com (Quanah Gibson-Mount) Date: Thu, 24 Apr 2014 16:37:31 -0700 Subject: [PATCH 1 of 2] HTTP: Add client source port to any error that is logged In-Reply-To: <20140424182607.GL34696@mdounin.ru> References: <4b7d2e503c06758330aa.1398359986@zre-ldap001.eng.zimbra.com> <20140424173753.GI34696@mdounin.ru> <861BE01750014CC67B7EF5A9@[192.168.1.15]> <20140424175648.GK34696@mdounin.ru> <20140424182607.GL34696@mdounin.ru> Message-ID: <35849A0D63680460F9814161@[192.168.1.15]> --On April 24, 2014 at 10:26:07 PM +0400 Maxim Dounin wrote: >> Yes, that is true, but why only implement a partial solution? With CGN, >> only logging the IP is fairly useless in all cases. To truly get useful >> information going forward, the IP + PORT of the client should logged in >> all cases. > > Access log certainly can be configured to provide enough > enformation to match any given error log message to a port if > needed. There is no need to implement anything, solution is > already here. The error log currently only provides the IP. While I'm guessing you could do things like correlate timestamps, it's still going to be a pain. Having the port readily available everywhere makes tracking a specific user much easier to do. > And, by asking about "why implement a partical solution" you are > overlooking the fact that proposed solution is partial as well - > it doesn't change c->addr_text to ensure proper logging in all > places (this would be a bad idea for other reasons, but it's > another question), but rather tries to hack on the http error > logging code to introduce remote port logging. This is far from > being a complete solution. I'm certainly willing to address any deficiencies, but I'd want to make sure it would follow whatever you want in the product before investing more time on it. ;) For now it meets the needs of our customer in Belgium who has to start dealing with the legal requirements of client port logging sooner than later. --Quanah -- Quanah Gibson-Mount Server Architect Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration From quanah at zimbra.com Thu Apr 24 23:38:45 2014 From: quanah at zimbra.com (Quanah Gibson-Mount) Date: Thu, 24 Apr 2014 16:38:45 -0700 Subject: [PATCH 2 of 2] MAIL: Always log the source port of the client In-Reply-To: <20140424174738.GJ34696@mdounin.ru> References: <3c908c40acd15c8df020.1398359987@zre-ldap001.eng.zimbra.com> <20140424174738.GJ34696@mdounin.ru> Message-ID: <967F87780C0CBEC34B650F87@[192.168.1.15]> --On April 24, 2014 at 9:47:39 PM +0400 Maxim Dounin wrote: > Hello! > > Much like http-related counterpart, this looks like a hack for me. > > We may consider adding port to the "client connected" messages > (not sure), but I don't think we have to do anything beyond that. For our end clients, who have to actually examine particular error messages, it is useful to have the port logged in any of the connection data. If the current solution is hackish, I'm happy to work out something more acceptable if you want to note what that would be. ;) --Quanah -- Quanah Gibson-Mount Server Architect Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration From aaron.peschel at gmail.com Fri Apr 25 01:20:24 2014 From: aaron.peschel at gmail.com (Aaron Peschel) Date: Thu, 24 Apr 2014 18:20:24 -0700 Subject: [nginx] Adding Support for Weak ETags In-Reply-To: <20140421122647.GB34696@mdounin.ru> References: <20140421122647.GB34696@mdounin.ru> Message-ID: Hi Maxim, Is the draft patch the same as the one that your posted in the previous thread, or has more work been done since then? -Aaron Peschel On Mon, Apr 21, 2014 at 5:26 AM, Maxim Dounin wrote: > Hello! > > On Thu, Apr 17, 2014 at 05:39:40PM -0700, Aaron Peschel wrote: > >> Hello, >> >> I am interested in getting support for Weak ETags into the mainline. >> There was some discussion previously in here previously that developed >> a quick patch to add support. What additional functionality would be >> required and what steps should be followed to get weak etag >> functionality added to nginx? I am willing to do the work, I just need >> some help with heading in the right direction. > > I had a quick draft patch sitting in my patchqueue since previous > discussion (see [1]) to downgrade strict etags to weak ones. It > needs more work though, as I'm not yet happy with the code. I > hope I'll be able to find some time and finish it in 1.7.x. > > [1] http://mailman.nginx.org/pipermail/nginx-devel/2013-November/004523.html > > -- > Maxim Dounin > http://nginx.org/ > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel From mp3geek at gmail.com Fri Apr 25 02:47:06 2014 From: mp3geek at gmail.com (Ryan Brown) Date: Fri, 25 Apr 2014 14:47:06 +1200 Subject: [PATCH] Configure: call "make distclean" for zlib only if Makefile exists In-Reply-To: References: Message-ID: Slightly off-topic, does cloudflare have a nginx fork publicly available? I only saw lua-nginx-cache-module On Thu, Apr 24, 2014 at 10:16 PM, Piotr Sikora wrote: > Hey, > > > Configure: call "make distclean" for zlib only if Makefile exists. > > > > This change allows to build nginx against git checkout of zlib. > > Actually, it looks that only our amd64-optimized version [1] doesn't > ship with Makefile, the original repository [2] has a minimal one with > "distclean" target, so feel free to ignore this patch (although, I > still think it's a good practice to check if the Makefile exists > before calling make). > > [1] https://github.com/cloudflare/zlib/commits/gcc.amd64 > [2] https://github.com/madler/zlib > > Best regards, > Piotr Sikora > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From agentzh at gmail.com Fri Apr 25 04:08:50 2014 From: agentzh at gmail.com (Yichun Zhang (agentzh)) Date: Thu, 24 Apr 2014 21:08:50 -0700 Subject: [PATCH] Configure: call "make distclean" for zlib only if Makefile exists In-Reply-To: References: Message-ID: Hello! On Thu, Apr 24, 2014 at 7:47 PM, Ryan Brown wrote: > Slightly off-topic, does cloudflare have a nginx fork publicly available? No. Actually we've been trying hard not to diverge from the official nginx core too far. That's why we've been working hard on the openresty bundle project: http://openresty.org > I > only saw lua-nginx-cache-module > This module is actually not really ready and we haven't used it yet in CloudFlare's online system. Regards, -agentzh From mdounin at mdounin.ru Fri Apr 25 11:35:58 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 25 Apr 2014 15:35:58 +0400 Subject: [nginx] Adding Support for Weak ETags In-Reply-To: References: <20140421122647.GB34696@mdounin.ru> Message-ID: <20140425113558.GN34696@mdounin.ru> Hello! On Thu, Apr 24, 2014 at 06:20:24PM -0700, Aaron Peschel wrote: > Hi Maxim, > > Is the draft patch the same as the one that your posted in the > previous thread, or has more work been done since then? The one I've posted is to ignore weak etags. The draft one is to downgrade strict etags to weak etags. > > -Aaron Peschel > > On Mon, Apr 21, 2014 at 5:26 AM, Maxim Dounin wrote: > > Hello! > > > > On Thu, Apr 17, 2014 at 05:39:40PM -0700, Aaron Peschel wrote: > > > >> Hello, > >> > >> I am interested in getting support for Weak ETags into the mainline. > >> There was some discussion previously in here previously that developed > >> a quick patch to add support. What additional functionality would be > >> required and what steps should be followed to get weak etag > >> functionality added to nginx? I am willing to do the work, I just need > >> some help with heading in the right direction. > > > > I had a quick draft patch sitting in my patchqueue since previous > > discussion (see [1]) to downgrade strict etags to weak ones. It > > needs more work though, as I'm not yet happy with the code. I > > hope I'll be able to find some time and finish it in 1.7.x. > > > > [1] http://mailman.nginx.org/pipermail/nginx-devel/2013-November/004523.html > > > > -- > > Maxim Dounin > > http://nginx.org/ > > > > _______________________________________________ > > nginx-devel mailing list > > nginx-devel at nginx.org > > http://mailman.nginx.org/mailman/listinfo/nginx-devel > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -- Maxim Dounin http://nginx.org/ From mdounin at mdounin.ru Fri Apr 25 11:49:42 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 25 Apr 2014 15:49:42 +0400 Subject: [PATCH 1 of 2] HTTP: Add client source port to any error that is logged In-Reply-To: <35849A0D63680460F9814161@[192.168.1.15]> References: <4b7d2e503c06758330aa.1398359986@zre-ldap001.eng.zimbra.com> <20140424173753.GI34696@mdounin.ru> <861BE01750014CC67B7EF5A9@[192.168.1.15]> <20140424175648.GK34696@mdounin.ru> <20140424182607.GL34696@mdounin.ru> <35849A0D63680460F9814161@[192.168.1.15]> Message-ID: <20140425114942.GP34696@mdounin.ru> Hello! On Thu, Apr 24, 2014 at 04:37:31PM -0700, Quanah Gibson-Mount wrote: > > > --On April 24, 2014 at 10:26:07 PM +0400 Maxim Dounin > wrote: > > >>Yes, that is true, but why only implement a partial solution? With CGN, > >>only logging the IP is fairly useless in all cases. To truly get useful > >>information going forward, the IP + PORT of the client should logged in > >>all cases. > > > >Access log certainly can be configured to provide enough > >enformation to match any given error log message to a port if > >needed. There is no need to implement anything, solution is > >already here. > > The error log currently only provides the IP. While I'm guessing you could > do things like correlate timestamps, it's still going to be a pain. Having > the port readily available everywhere makes tracking a specific user much > easier to do. Each error log message contains connection id. > >And, by asking about "why implement a partical solution" you are > >overlooking the fact that proposed solution is partial as well - > >it doesn't change c->addr_text to ensure proper logging in all > >places (this would be a bad idea for other reasons, but it's > >another question), but rather tries to hack on the http error > >logging code to introduce remote port logging. This is far from > >being a complete solution. > > I'm certainly willing to address any deficiencies, but I'd want to make sure > it would follow whatever you want in the product before investing more time > on it. ;) For now it meets the needs of our customer in Belgium who has to > start dealing with the legal requirements of client port logging sooner than > later. Fair enough. As I already said in my first messages, I tend to think that there are no changes needed here. -- Maxim Dounin http://nginx.org/ From john at disqus.com Fri Apr 25 19:47:28 2014 From: john at disqus.com (John Watson) Date: Fri, 25 Apr 2014 12:47:28 -0700 Subject: Help with shared memory usage In-Reply-To: <20140122165150.GP1835@mdounin.ru> References: <20130729171109.GA2130@mdounin.ru> <20130730100931.GD2130@mdounin.ru> <20131006093708.GY62063@mdounin.ru> <20131220164923.GK95113@mdounin.ru> <20140122165150.GP1835@mdounin.ru> Message-ID: Hi Maxim, Any chance you can fix up this patch and get it into 1.7? This patch is not compatible with 1.6. Best, John On Wed, Jan 22, 2014 at 8:51 AM, Maxim Dounin wrote: > Hello! > > On Wed, Jan 22, 2014 at 01:39:54AM -0200, Wandenberg Peixoto wrote: > >> Hello Maxim, >> >> did you have opportunity to take a look on this last patch? > > It looks more or less correct, though I don't happy with the > checks done, and there are various style issues. I'm planning to > look into it and build a better version as time permits. > >> >> >> Regards, >> Wandenberg >> >> >> On Thu, Dec 26, 2013 at 10:12 PM, Wandenberg Peixoto >> wrote: >> >> > Hello Maxim, >> > >> > I changed the patch to check only the p->next pointer. >> > And checking if the page is in an address less than the (pool->pages + >> > pages). >> > >> > + ngx_slab_page_t *prev, *p; >> > + ngx_uint_t pages; >> > + size_t size; >> > + >> > + size = pool->end - (u_char *) pool - sizeof(ngx_slab_pool_t); >> > + pages = (ngx_uint_t) (size / (ngx_pagesize + >> > sizeof(ngx_slab_page_t))); >> > + >> > + p = &page[page->slab]; >> > + >> > + if ((p < pool->pages + pages) && >> > + (p->next != NULL) && >> > + ((p->prev & NGX_SLAB_PAGE_MASK) == NGX_SLAB_PAGE)) { >> > >> > >> > I hope that now I did the right checks. >> > >> > Regards, >> > Wandenberg >> > >> > >> > On Fri, Dec 20, 2013 at 2:49 PM, Maxim Dounin wrote: >> > >> >> Hello! >> >> >> >> On Tue, Dec 17, 2013 at 09:05:16PM -0200, Wandenberg Peixoto wrote: >> >> >> >> > Hi Maxim, >> >> > >> >> > sorry for the long delay. I hope you remember my issue. >> >> > In attach is the new patch with the changes you suggest. >> >> > Can you check it again? I hope it can be applied to nginx code now. >> >> > >> >> > About this point "2. There is probably no need to check both prev and >> >> > next.", >> >> > I check both pointers to avoid a segmentation fault, since in some >> >> > situations the next can be NULL and the prev may point to pool->free. >> >> > As far as I could follow the code seems to me that could happen one of >> >> this >> >> > pointers, next or prev, point to a NULL. >> >> > I just made a double check to protect. >> >> >> >> As far as I see, all pages in the pool->free list are expected to >> >> have both p->prev and p->next set. And all pages with type >> >> NGX_SLAB_PAGE will be either on the pool->free list, or will have >> >> p->next set to NULL. >> >> >> >> [...] >> >> >> >> > > > +{ >> >> > > > + ngx_slab_page_t *neighbour = &page[page->slab]; >> >> > > >> >> > > Here "neighbour" may point outside of the allocated page >> >> > > structures if we are freeing the last page in the pool. >> >> >> >> It looks like you've tried to address this problem with the >> >> following check: >> >> >> >> > +static ngx_int_t >> >> > +ngx_slab_merge_pages(ngx_slab_pool_t *pool, ngx_slab_page_t *page) >> >> > +{ >> >> > + ngx_slab_page_t *prev, *p; >> >> > + >> >> > + p = &page[page->slab]; >> >> > + if ((u_char *) p >= pool->end) { >> >> > + return NGX_DECLINED; >> >> > + } >> >> >> >> This looks wrong, as pool->end points to the end of the pool, not >> >> the end of allocated page structures. >> >> >> >> -- >> >> Maxim Dounin >> >> http://nginx.org/ >> >> >> >> _______________________________________________ >> >> nginx-devel mailing list >> >> nginx-devel at nginx.org >> >> http://mailman.nginx.org/mailman/listinfo/nginx-devel >> >> >> > >> > > >> _______________________________________________ >> nginx-devel mailing list >> nginx-devel at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > -- > Maxim Dounin > http://nginx.org/ > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel From fdasilvayy at gmail.com Sat Apr 26 11:05:52 2014 From: fdasilvayy at gmail.com (Filipe Da Silva) Date: Sat, 26 Apr 2014 13:05:52 +0200 Subject: Help with shared memory usage In-Reply-To: References: <20130729171109.GA2130@mdounin.ru> <20130730100931.GD2130@mdounin.ru> <20131006093708.GY62063@mdounin.ru> <20131220164923.GK95113@mdounin.ru> <20140122165150.GP1835@mdounin.ru> Message-ID: Hello,, AFAIK, the patch attached here : http://mailman.nginx.org/pipermail/nginx-devel/2013-December/004721.html compile, and should work on 1.6 too. Maybe the merge is a bit tricky with the last changes made on ngx_slab.c I have a up-to-date version on my patch queue. as Maxim stated, it still needs a functional rework . http://mailman.nginx.org/pipermail/nginx-devel/2013-October/004337.html Best, Filipe 2014-04-25 21:47 GMT+02:00 John Watson : > Hi Maxim, > > Any chance you can fix up this patch and get it into 1.7? > > This patch is not compatible with 1.6. > > Best, > > John > > On Wed, Jan 22, 2014 at 8:51 AM, Maxim Dounin wrote: >> Hello! >> >> On Wed, Jan 22, 2014 at 01:39:54AM -0200, Wandenberg Peixoto wrote: >> >>> Hello Maxim, >>> >>> did you have opportunity to take a look on this last patch? >> >> It looks more or less correct, though I don't happy with the >> checks done, and there are various style issues. I'm planning to >> look into it and build a better version as time permits. >> >>> >>> >>> Regards, >>> Wandenberg >>> >>> >>> On Thu, Dec 26, 2013 at 10:12 PM, Wandenberg Peixoto >>> wrote: >>> >>> > Hello Maxim, >>> > >>> > I changed the patch to check only the p->next pointer. >>> > And checking if the page is in an address less than the (pool->pages + >>> > pages). >>> > >>> > + ngx_slab_page_t *prev, *p; >>> > + ngx_uint_t pages; >>> > + size_t size; >>> > + >>> > + size = pool->end - (u_char *) pool - sizeof(ngx_slab_pool_t); >>> > + pages = (ngx_uint_t) (size / (ngx_pagesize + >>> > sizeof(ngx_slab_page_t))); >>> > + >>> > + p = &page[page->slab]; >>> > + >>> > + if ((p < pool->pages + pages) && >>> > + (p->next != NULL) && >>> > + ((p->prev & NGX_SLAB_PAGE_MASK) == NGX_SLAB_PAGE)) { >>> > >>> > >>> > I hope that now I did the right checks. >>> > >>> > Regards, >>> > Wandenberg >>> > >>> > >>> > On Fri, Dec 20, 2013 at 2:49 PM, Maxim Dounin wrote: >>> > >>> >> Hello! >>> >> >>> >> On Tue, Dec 17, 2013 at 09:05:16PM -0200, Wandenberg Peixoto wrote: >>> >> >>> >> > Hi Maxim, >>> >> > >>> >> > sorry for the long delay. I hope you remember my issue. >>> >> > In attach is the new patch with the changes you suggest. >>> >> > Can you check it again? I hope it can be applied to nginx code now. >>> >> > >>> >> > About this point "2. There is probably no need to check both prev and >>> >> > next.", >>> >> > I check both pointers to avoid a segmentation fault, since in some >>> >> > situations the next can be NULL and the prev may point to pool->free. >>> >> > As far as I could follow the code seems to me that could happen one of >>> >> this >>> >> > pointers, next or prev, point to a NULL. >>> >> > I just made a double check to protect. >>> >> >>> >> As far as I see, all pages in the pool->free list are expected to >>> >> have both p->prev and p->next set. And all pages with type >>> >> NGX_SLAB_PAGE will be either on the pool->free list, or will have >>> >> p->next set to NULL. >>> >> >>> >> [...] >>> >> >>> >> > > > +{ >>> >> > > > + ngx_slab_page_t *neighbour = &page[page->slab]; >>> >> > > >>> >> > > Here "neighbour" may point outside of the allocated page >>> >> > > structures if we are freeing the last page in the pool. >>> >> >>> >> It looks like you've tried to address this problem with the >>> >> following check: >>> >> >>> >> > +static ngx_int_t >>> >> > +ngx_slab_merge_pages(ngx_slab_pool_t *pool, ngx_slab_page_t *page) >>> >> > +{ >>> >> > + ngx_slab_page_t *prev, *p; >>> >> > + >>> >> > + p = &page[page->slab]; >>> >> > + if ((u_char *) p >= pool->end) { >>> >> > + return NGX_DECLINED; >>> >> > + } >>> >> >>> >> This looks wrong, as pool->end points to the end of the pool, not >>> >> the end of allocated page structures. >>> >> >>> >> -- >>> >> Maxim Dounin >>> >> http://nginx.org/ >>> >> >>> >> _______________________________________________ >>> >> nginx-devel mailing list >>> >> nginx-devel at nginx.org >>> >> http://mailman.nginx.org/mailman/listinfo/nginx-devel >>> >> >>> > >>> > >> >>> _______________________________________________ >>> nginx-devel mailing list >>> nginx-devel at nginx.org >>> http://mailman.nginx.org/mailman/listinfo/nginx-devel >> >> >> -- >> Maxim Dounin >> http://nginx.org/ >> >> _______________________________________________ >> nginx-devel mailing list >> nginx-devel at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel From pug at felsing.net Sun Apr 27 07:54:35 2014 From: pug at felsing.net (Christian Felsing) Date: Sun, 27 Apr 2014 09:54:35 +0200 Subject: [PATCH] Mail: added support for SSL client certificate Message-ID: <535CB7BB.3080800@felsing.net> Hello, this patch has an buffer length calculation issue in src/mail/ngx_mail_auth_http_module.c, in case of multiple login - logout sequences sometimes I got signal 11 errors in log which are caused by memory access outside that buffer. This may also a security issue. len = sizeof("GET ") - 1 + ahcf->uri.len + sizeof(" HTTP/1.0" CRLF) - 1 + sizeof("Host: ") - 1 + ahcf->host_header.len + sizeof(CRLF) - 1 + sizeof("Auth-Method: ") - 1 + ngx_mail_auth_http_method[s->auth_method].len + sizeof(CRLF) - 1 + sizeof("Auth-User: ") - 1 + login.len + sizeof(CRLF) - 1 + sizeof("Auth-Pass: ") - 1 + passwd.len + sizeof(CRLF) - 1 + sizeof("Auth-Salt: ") - 1 + s->salt.len #if (NGX_MAIL_SSL) + sizeof("Auth-Certificate: ") - 1 + cert.len + sizeof(CRLF) - 1 + sizeof("Auth-Verify: ") - 1 + verify.len + sizeof(CRLF) - 1 + sizeof("Auth-Issuer-DN: ") - 1 + issuer.len + sizeof(CRLF) - 1 + sizeof("Auth-Subject-DN: ") - 1 + subject.len + sizeof(CRLF) - 1 + sizeof("Auth-Subject-Serial: ") - 1 + serial.len + sizeof(CRLF) - 1 #endif + sizeof("Auth-Protocol: ") - 1 + cscf->protocol->name.len + sizeof(CRLF) - 1 + sizeof("Auth-Login-Attempt: ") - 1 + NGX_INT_T_LEN + sizeof(CRLF) - 1 + sizeof("Client-IP: ") - 1 + s->connection->addr_text.len + sizeof(CRLF) - 1 + sizeof("Client-Host: ") - 1 + s->host.len + sizeof(CRLF) - 1 + sizeof("Auth-SMTP-Helo: ") - 1 + s->smtp_helo.len + sizeof("Auth-SMTP-From: ") - 1 + s->smtp_from.len + sizeof("Auth-SMTP-To: ") - 1 + s->smtp_to.len + ahcf->header.len + sizeof(CRLF) - 1 + 20; // <====================== To verify that I added 20 extra byte to buffer which fixed that problem in this special case - this dirty patch was only for verification. I think buffer length calculation is wrong. Further questions arises: * What happens if subjects (issuer or other attribute) contains 0x00? Do functions like ngx_ssl_get_issuer_dn care about that? * All Attributes may contain UTF-8 characters, that means a character may consist of 1 to 4 bytes. Does length calculation care about that? before doing something like b->last = ngx_cpymem(b->last, "GET ", sizeof("GET ") - 1); it should be checked if ((ngx_buf_t *)(b->last + (sizeof("GET ") - 1)) > (b + len)) { ngx_log_error(NGX_LOG_ALERT, s->connection->log, 0, "buffer size exceeded"); return NULL; } best regards Christian Felsing From fdasilvayy at gmail.com Sun Apr 27 10:53:51 2014 From: fdasilvayy at gmail.com (Filipe Da Silva) Date: Sun, 27 Apr 2014 12:53:51 +0200 Subject: [PATCH] Mail: added support for SSL client certificate In-Reply-To: <535CB7BB.3080800@felsing.net> References: <535CB7BB.3080800@felsing.net> Message-ID: Hi, My answer are below. 2014-04-27 9:54 GMT+02:00 Christian Felsing : > Hello, > > this patch has an buffer length calculation issue in > src/mail/ngx_mail_auth_http_module.c, in case of multiple login - logout > sequences sometimes I got signal 11 errors in log which are caused by > memory access outside that buffer. This may also a security issue. > > len = sizeof("GET ") - 1 + ahcf->uri.len + sizeof(" HTTP/1.0" CRLF) - 1 > + sizeof("Host: ") - 1 + ahcf->host_header.len + sizeof(CRLF) - 1 > + sizeof("Auth-Method: ") - 1 > + ngx_mail_auth_http_method[s->auth_method].len > + sizeof(CRLF) - 1 > + sizeof("Auth-User: ") - 1 + login.len + sizeof(CRLF) - 1 > + sizeof("Auth-Pass: ") - 1 + passwd.len + sizeof(CRLF) - 1 > + sizeof("Auth-Salt: ") - 1 + s->salt.len > #if (NGX_MAIL_SSL) > + sizeof("Auth-Certificate: ") - 1 + cert.len + sizeof(CRLF) - 1 > + sizeof("Auth-Verify: ") - 1 + verify.len + sizeof(CRLF) - 1 > + sizeof("Auth-Issuer-DN: ") - 1 + issuer.len + sizeof(CRLF) - 1 > + sizeof("Auth-Subject-DN: ") - 1 + subject.len + sizeof(CRLF) - 1 > + sizeof("Auth-Subject-Serial: ") - 1 + serial.len > + sizeof(CRLF) - 1 > #endif > + sizeof("Auth-Protocol: ") - 1 + cscf->protocol->name.len > + sizeof(CRLF) - 1 > + sizeof("Auth-Login-Attempt: ") - 1 + NGX_INT_T_LEN > + sizeof(CRLF) - 1 > + sizeof("Client-IP: ") - 1 + s->connection->addr_text.len > + sizeof(CRLF) - 1 > + sizeof("Client-Host: ") - 1 + s->host.len + sizeof(CRLF) - 1 > + sizeof("Auth-SMTP-Helo: ") - 1 + s->smtp_helo.len > + sizeof("Auth-SMTP-From: ") - 1 + s->smtp_from.len > + sizeof("Auth-SMTP-To: ") - 1 + s->smtp_to.len > + ahcf->header.len > + sizeof(CRLF) - 1 > + 20; // <====================== > > > To verify that I added 20 extra byte to buffer which fixed that problem > in this special case - this dirty patch was only for verification. I > think buffer length calculation is wrong. Further questions arises: > I'm not sure about this, as this buffer already has an oversized allocation: The SMTP header len is added whatever the current mail protocol, for example. > * What happens if subjects (issuer or other attribute) contains 0x00? Do > functions like ngx_ssl_get_issuer_dn care about that? > > * All Attributes may contain UTF-8 characters, that means a character > may consist of 1 to 4 bytes. Does length calculation care about that? > The code of ngx_ssl_get_issuer_dn contains a loop that recalc the exact text len : for (len = 0; p[len]; len++) { /* void */ } Same about the certificate subject. C strings are zero terminated, whatever its contains : UTF-8 or not. > before doing something like > b->last = ngx_cpymem(b->last, "GET ", sizeof("GET ") - 1); > it should be checked > > if ((ngx_buf_t *)(b->last + (sizeof("GET ") - 1)) > (b + len)) { > ngx_log_error(NGX_LOG_ALERT, s->connection->log, 0, > "buffer size exceeded"); > return NULL; > } Please try this patch : ==== # HG changeset patch # User Filipe da Silva # Date 1398595855 -7200 # Sun Apr 27 12:50:55 2014 +0200 # Node ID d20159ce27d7f13813178c820c4c74da14b38e11 # Parent 741297fe4dc26d04deccd7bbcb1dc53c18f56614 Mail: http request buffer overflow check diff -r 741297fe4dc2 -r d20159ce27d7 src/mail/ngx_mail_auth_http_module.c --- a/src/mail/ngx_mail_auth_http_module.c Fri Jan 24 16:26:16 2014 +0100 +++ b/src/mail/ngx_mail_auth_http_module.c Sun Apr 27 12:50:55 2014 +0200 @@ -1381,6 +1381,13 @@ ngx_mail_auth_http_create_request(ngx_ma /* add "\r\n" at the header end */ *b->last++ = CR; *b->last++ = LF; + if ( b->last > b->end ) + { + ngx_log_error(NGX_LOG_EMERG, s->connection->log, 0, + "Buffer overflow while creating http request: " + "%uz used instead of %uz", b->last - b->start, len); + } + #if (NGX_DEBUG_MAIL_PASSWD) { ngx_str_t l; ==== It will directly check if there is any buffer overflow. > best regards > Christian Felsing > Best, Filipe da Silva From pug at felsing.net Sun Apr 27 13:04:06 2014 From: pug at felsing.net (Christian Felsing) Date: Sun, 27 Apr 2014 15:04:06 +0200 Subject: [PATCH] Mail: added support for SSL client certificate In-Reply-To: References: <535CB7BB.3080800@felsing.net> Message-ID: <535D0046.4010305@felsing.net> Hi, Am 27.04.2014 12:53, schrieb Filipe Da Silva: > I'm not sure about this, as this buffer already has an oversized allocation: ...I hope so... > Same about the certificate subject. > C strings are zero terminated, whatever its contains : UTF-8 or not. if all used libraries do so, this should be ok. > Please try this patch : > It will directly check if there is any buffer overflow. I will try that, but reproduction is difficult, I got sometimes that Signal 11 with Thunderbird as client and Dovecot 2.1.12 as IMAP backend, where Nginx talks via port 143 with Dovecot. May Perl auth script able to generate such effects? best regards Christian ---My Nginx nginx-1.7.0 built on Debian 7.4 - 64 bit (latest updates) ./configure --with-file-aio --with-ipv6 --with-openssl=../openssl-1.0.1g --with-openssl-opt="fips shared -g -march=native -DOPENSSL_NO_HEARTBEATS" --with-mail --with-mail_ssl_module --with-http_ssl_module --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx/nginx.pid --lock-path=/var/lock/nginx/nginx.lock --with-http_perl_module Perl script for auth (/usr/local/nginx/perl/lib/mailauth.pm): package mailauth; # from # http://wiki.nginx.org/ImapAuthenticateWithEmbeddedPerlScript use strict; use nginx; my $DEBUG=1; our $auth_ok; our $mail_server_ip={}; our $protocol_ports={}; $mail_server_ip->{'mailhost01'}="192.168.42.1"; $protocol_ports->{'pop3'}=110; $protocol_ports->{'imap'}=143; sub handler { my $r = shift; # security depends only on Nginx certificate verification # configuration. You may add additional verification here. # If Nginx comes to here, it already verified users # client certificate. # YOU MUST SET ssl_verify_client on; OTHERWISE WHOLE # WORLD CAN ACCESS ALL YOUR MAILS. my @subject=split(/\//,$r->header_in("Auth-Subject-DN")); my $emailAddress=undef; foreach (@subject) { if ($_=~/^emailAddress=(\S+)/) { $emailAddress="$1"; } } # Care about broken client certs if (!defined($emailAddress)) { return DECLINED; } $auth_ok=1; # Think twice about your Nginx config if ($auth_ok==1){ $r->header_out("Auth-Status", "OK") ; $r->header_out("Auth-Server", $mail_server_ip->{'mailhost01'}); $r->header_out("Auth-Port", $protocol_ports->{$r->header_in("Auth-Protocol")}); $r->header_out("Auth-User", "$emailAddress"); $r->header_out("Auth-Pass", "Some arbitrary password where Dovecot does not care about"); # Nginx: Talk to me if ($DEBUG) { $r->log_error(0, "emailAddress: $emailAddress"); $r->log_error(0, "Auth-Server: " . $mail_server_ip->{'mailhost01'}); $r->log_error(0, "Auth-Protocol: " . $r->header_in("Auth-Protocol")); $r->log_error(0, "Auth-Port: " . $protocol_ports->{$r->header_in("Auth-Protocol")}); $r->log_error(0, "Auth-User: " . "$emailAddress"); } } else { $r->header_out("Auth-Status", "Invalid login or password") ; } $r->send_http_header("text/html"); return OK; } 1; __END__ From pug at felsing.net Sun Apr 27 13:22:30 2014 From: pug at felsing.net (Christian Felsing) Date: Sun, 27 Apr 2014 15:22:30 +0200 Subject: [PATCH] Mail: added support for SSL client certificate In-Reply-To: <535D0046.4010305@felsing.net> References: <535CB7BB.3080800@felsing.net> <535D0046.4010305@felsing.net> Message-ID: <535D0496.4040604@felsing.net> one more result. If DEBUG is 0 in Perl script, Signal 11 occurs, if DEBUG=1 everything works. Christian From info at kliemeck.de Sun Apr 27 20:10:12 2014 From: info at kliemeck.de (info at kliemeck.de) Date: Sun, 27 Apr 2014 22:10:12 +0200 Subject: [PATCH] Added nonlocal to the listen directive In-Reply-To: <20140328122528.GL34696@mdounin.ru> References: <16eacd8609c8362e9dd7.1395999953@miitool> <20140328122528.GL34696@mdounin.ru> Message-ID: <20140427221012.Horde.pUFTrYPMwbaBfz1QpJpkjA1@webmail.your-server.de> Hey, but it is still not possible to work with IPv6, if you want to bind to a specific address (not [::]) that is not a local address. The "ip_nonlocal_bind-sysctl" use-case is not fulfilled with this and i think it is a common use-case that nginx is used within a high availability environment with a shared ip address. It is possible that this important feature is integrated within 1.6, since it may be a reason not to use IPv6? greets Hans-Joachim Kliemeck Quoting mdounin at mdounin.ru: > Hello! > > On Fri, Mar 28, 2014 at 10:45:53AM +0100, Trygve Vea wrote: > >> # HG changeset patch >> # User Trygve Vea >> # Date 1395999940 -3600 >> # Fri Mar 28 10:45:40 2014 +0100 >> # Node ID 16eacd8609c8362e9dd729c743ed7a869c2993fe >> # Parent 2411d4b5be2ca690a5a00a1d8ad96ff69a00317f >> Added nonlocal to the listen directive >> >> The nonlocal option is used to set the needed socket options to be >> able to bind >> to an address not necessarily owned by the host. >> >> This patch currently implements this for Linux >= 2.4 IPv4/IPv6. >> >> The problem we solve by doing this, is in an environment where the following >> conditions are met: >> >> * HTTPS with multiple certificates, and a client base that are unable to use >> SNI - thus having the need to tie specific certificates to >> specific ip/ports. >> * Setting the ip_nonlocal_bind-sysctl is not an option (for example >> for Linux >> IPv6) >> * Used in a failover-setup, where the service IP-addresses are >> moved around by >> a daemon like linux-ha or keepalived. > > As already explained, the patch is not needed for the use case > claimed. Just a bind on INADDR_ANY/IN6ADDR_ANY will do the trick. > > -- > Maxim Dounin > http://nginx.org/ From elvis.deng at ericsson.com Mon Apr 28 04:59:54 2014 From: elvis.deng at ericsson.com (Elvis Deng) Date: Mon, 28 Apr 2014 04:59:54 +0000 Subject: How to modify a POST arg within Nginx? Message-ID: <5E5CDFB4A43EA8419ECBA8D40485C9D81C756D46@ESGSCMB107.ericsson.se> Hi all, I want to modify a POST value in Nginx and then put it back (don't change its original data flow). Does any existing module do the similar work? Thanks. Best wishes, Elvis Deng -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Mon Apr 28 12:11:20 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 28 Apr 2014 16:11:20 +0400 Subject: Help with shared memory usage In-Reply-To: References: <20130730100931.GD2130@mdounin.ru> <20131006093708.GY62063@mdounin.ru> <20131220164923.GK95113@mdounin.ru> <20140122165150.GP1835@mdounin.ru> Message-ID: <20140428121119.GY34696@mdounin.ru> Hello! On Fri, Apr 25, 2014 at 12:47:28PM -0700, John Watson wrote: > Hi Maxim, > > Any chance you can fix up this patch and get it into 1.7? It's still in my queue, I hope I can spend some time on it in the near future. (It's relatively low priority though, as nothing in nginx itself uses shared memory allocations larger than page size.) > > This patch is not compatible with 1.6. > > Best, > > John > > On Wed, Jan 22, 2014 at 8:51 AM, Maxim Dounin wrote: > > Hello! > > > > On Wed, Jan 22, 2014 at 01:39:54AM -0200, Wandenberg Peixoto wrote: > > > >> Hello Maxim, > >> > >> did you have opportunity to take a look on this last patch? > > > > It looks more or less correct, though I don't happy with the > > checks done, and there are various style issues. I'm planning to > > look into it and build a better version as time permits. > > > >> > >> > >> Regards, > >> Wandenberg > >> > >> > >> On Thu, Dec 26, 2013 at 10:12 PM, Wandenberg Peixoto > >> wrote: > >> > >> > Hello Maxim, > >> > > >> > I changed the patch to check only the p->next pointer. > >> > And checking if the page is in an address less than the (pool->pages + > >> > pages). > >> > > >> > + ngx_slab_page_t *prev, *p; > >> > + ngx_uint_t pages; > >> > + size_t size; > >> > + > >> > + size = pool->end - (u_char *) pool - sizeof(ngx_slab_pool_t); > >> > + pages = (ngx_uint_t) (size / (ngx_pagesize + > >> > sizeof(ngx_slab_page_t))); > >> > + > >> > + p = &page[page->slab]; > >> > + > >> > + if ((p < pool->pages + pages) && > >> > + (p->next != NULL) && > >> > + ((p->prev & NGX_SLAB_PAGE_MASK) == NGX_SLAB_PAGE)) { > >> > > >> > > >> > I hope that now I did the right checks. > >> > > >> > Regards, > >> > Wandenberg > >> > > >> > > >> > On Fri, Dec 20, 2013 at 2:49 PM, Maxim Dounin wrote: > >> > > >> >> Hello! > >> >> > >> >> On Tue, Dec 17, 2013 at 09:05:16PM -0200, Wandenberg Peixoto wrote: > >> >> > >> >> > Hi Maxim, > >> >> > > >> >> > sorry for the long delay. I hope you remember my issue. > >> >> > In attach is the new patch with the changes you suggest. > >> >> > Can you check it again? I hope it can be applied to nginx code now. > >> >> > > >> >> > About this point "2. There is probably no need to check both prev and > >> >> > next.", > >> >> > I check both pointers to avoid a segmentation fault, since in some > >> >> > situations the next can be NULL and the prev may point to pool->free. > >> >> > As far as I could follow the code seems to me that could happen one of > >> >> this > >> >> > pointers, next or prev, point to a NULL. > >> >> > I just made a double check to protect. > >> >> > >> >> As far as I see, all pages in the pool->free list are expected to > >> >> have both p->prev and p->next set. And all pages with type > >> >> NGX_SLAB_PAGE will be either on the pool->free list, or will have > >> >> p->next set to NULL. > >> >> > >> >> [...] > >> >> > >> >> > > > +{ > >> >> > > > + ngx_slab_page_t *neighbour = &page[page->slab]; > >> >> > > > >> >> > > Here "neighbour" may point outside of the allocated page > >> >> > > structures if we are freeing the last page in the pool. > >> >> > >> >> It looks like you've tried to address this problem with the > >> >> following check: > >> >> > >> >> > +static ngx_int_t > >> >> > +ngx_slab_merge_pages(ngx_slab_pool_t *pool, ngx_slab_page_t *page) > >> >> > +{ > >> >> > + ngx_slab_page_t *prev, *p; > >> >> > + > >> >> > + p = &page[page->slab]; > >> >> > + if ((u_char *) p >= pool->end) { > >> >> > + return NGX_DECLINED; > >> >> > + } > >> >> > >> >> This looks wrong, as pool->end points to the end of the pool, not > >> >> the end of allocated page structures. > >> >> > >> >> -- > >> >> Maxim Dounin > >> >> http://nginx.org/ > >> >> > >> >> _______________________________________________ > >> >> nginx-devel mailing list > >> >> nginx-devel at nginx.org > >> >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > >> >> > >> > > >> > > > > >> _______________________________________________ > >> nginx-devel mailing list > >> nginx-devel at nginx.org > >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > > > > -- > > Maxim Dounin > > http://nginx.org/ > > > > _______________________________________________ > > nginx-devel mailing list > > nginx-devel at nginx.org > > http://mailman.nginx.org/mailman/listinfo/nginx-devel > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -- Maxim Dounin http://nginx.org/ From mdounin at mdounin.ru Mon Apr 28 12:52:12 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 28 Apr 2014 16:52:12 +0400 Subject: How to modify a POST arg within Nginx? In-Reply-To: <5E5CDFB4A43EA8419ECBA8D40485C9D81C756D46@ESGSCMB107.ericsson.se> References: <5E5CDFB4A43EA8419ECBA8D40485C9D81C756D46@ESGSCMB107.ericsson.se> Message-ID: <20140428125211.GA34696@mdounin.ru> Hello! On Mon, Apr 28, 2014 at 04:59:54AM +0000, Elvis Deng wrote: > Hi all, > > I want to modify a POST value in Nginx and then put it back (don't change its original data flow). > Does any existing module do the similar work? Thanks. For simple use cases, the proxy_set_body directive can be used, see http://nginx.org/r/proxy_set_body. Some 3rd party modules try to read request body themself and/or read it and then restart request processing with internal redirect or something like, but this is more or less a hack. An API for a generic request body modifications isn't yet here, though there is an experminental patch previously published, see here: http://mailman.nginx.org/pipermail/nginx-devel/2013-March/003492.html -- Maxim Dounin http://nginx.org/ From 12784453 at qq.com Tue Apr 29 12:45:34 2014 From: 12784453 at qq.com (=?gb18030?B?udrKpA==?=) Date: Tue, 29 Apr 2014 20:45:34 +0800 Subject: how to add a module to decode/modify parameter on HTTP request Message-ID: Hello, We are new on Nginx module development. Currently we encounter an issue as below. Could you please help to check? Thanks in advanced. Scenario: add a module to decode/modify parameter on HTTP request. Then pass the updated HTTP request to PHP to be handled as normal. Question: How to modify nginx.conf and how to decode/modify HTTP request? Currently, all examples about nginx module I got are nginx modules handle HTTP request and then provide response to client directly. nginx.conf ... location ~ \.php$ { fastcgi_split_path_info ^(.+\.php)(/.+)$; # NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini # With php5-cgi alone: fastcgi_pass 127.0.0.1:9000; # With php5-fpm: fastcgi_pass unix:/var/run/php5-fpm.sock; fastcgi_index index.php; include fastcgi_params; } -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.peschel at gmail.com Tue Apr 29 18:47:46 2014 From: aaron.peschel at gmail.com (Aaron Peschel) Date: Tue, 29 Apr 2014 11:47:46 -0700 Subject: [nginx] Adding Support for Weak ETags In-Reply-To: <20140425113558.GN34696@mdounin.ru> References: <20140421122647.GB34696@mdounin.ru> <20140425113558.GN34696@mdounin.ru> Message-ID: Hello Maxim, If you provide a copy of your newer draft patch, I am willing to spend time helping improve it as you see fit. -Aaron Peschel On Fri, Apr 25, 2014 at 4:35 AM, Maxim Dounin wrote: > Hello! > > On Thu, Apr 24, 2014 at 06:20:24PM -0700, Aaron Peschel wrote: > >> Hi Maxim, >> >> Is the draft patch the same as the one that your posted in the >> previous thread, or has more work been done since then? > > The one I've posted is to ignore weak etags. The draft one is to > downgrade strict etags to weak etags. > >> >> -Aaron Peschel >> >> On Mon, Apr 21, 2014 at 5:26 AM, Maxim Dounin wrote: >> > Hello! >> > >> > On Thu, Apr 17, 2014 at 05:39:40PM -0700, Aaron Peschel wrote: >> > >> >> Hello, >> >> >> >> I am interested in getting support for Weak ETags into the mainline. >> >> There was some discussion previously in here previously that developed >> >> a quick patch to add support. What additional functionality would be >> >> required and what steps should be followed to get weak etag >> >> functionality added to nginx? I am willing to do the work, I just need >> >> some help with heading in the right direction. >> > >> > I had a quick draft patch sitting in my patchqueue since previous >> > discussion (see [1]) to downgrade strict etags to weak ones. It >> > needs more work though, as I'm not yet happy with the code. I >> > hope I'll be able to find some time and finish it in 1.7.x. >> > >> > [1] http://mailman.nginx.org/pipermail/nginx-devel/2013-November/004523.html >> > >> > -- >> > Maxim Dounin >> > http://nginx.org/ >> > >> > _______________________________________________ >> > nginx-devel mailing list >> > nginx-devel at nginx.org >> > http://mailman.nginx.org/mailman/listinfo/nginx-devel >> >> _______________________________________________ >> nginx-devel mailing list >> nginx-devel at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > > -- > Maxim Dounin > http://nginx.org/ > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel From adam.arsenault at hootsuite.com Tue Apr 29 18:53:35 2014 From: adam.arsenault at hootsuite.com (Adam Arsenault) Date: Tue, 29 Apr 2014 11:53:35 -0700 Subject: [nginx] Adding Support for Weak ETags In-Reply-To: References: <20140421122647.GB34696@mdounin.ru> <20140425113558.GN34696@mdounin.ru> Message-ID: Hi Maxim/Aaron, Would love to see support for Weak Etags in nginx. The lack of support for gzip + etags has been a major problem for us as we use etags for caching (and need gzip to work with it) in a bunch of different place in our application. Let me know if there is anything I can do to help out here as well. Thanks, Adam Arsenault On Tue, Apr 29, 2014 at 11:47 AM, Aaron Peschel wrote: > Hello Maxim, > > If you provide a copy of your newer draft patch, I am willing to spend > time helping improve it as you see fit. > > -Aaron Peschel > > On Fri, Apr 25, 2014 at 4:35 AM, Maxim Dounin wrote: > > Hello! > > > > On Thu, Apr 24, 2014 at 06:20:24PM -0700, Aaron Peschel wrote: > > > >> Hi Maxim, > >> > >> Is the draft patch the same as the one that your posted in the > >> previous thread, or has more work been done since then? > > > > The one I've posted is to ignore weak etags. The draft one is to > > downgrade strict etags to weak etags. > > > >> > >> -Aaron Peschel > >> > >> On Mon, Apr 21, 2014 at 5:26 AM, Maxim Dounin > wrote: > >> > Hello! > >> > > >> > On Thu, Apr 17, 2014 at 05:39:40PM -0700, Aaron Peschel wrote: > >> > > >> >> Hello, > >> >> > >> >> I am interested in getting support for Weak ETags into the mainline. > >> >> There was some discussion previously in here previously that > developed > >> >> a quick patch to add support. What additional functionality would be > >> >> required and what steps should be followed to get weak etag > >> >> functionality added to nginx? I am willing to do the work, I just > need > >> >> some help with heading in the right direction. > >> > > >> > I had a quick draft patch sitting in my patchqueue since previous > >> > discussion (see [1]) to downgrade strict etags to weak ones. It > >> > needs more work though, as I'm not yet happy with the code. I > >> > hope I'll be able to find some time and finish it in 1.7.x. > >> > > >> > [1] > http://mailman.nginx.org/pipermail/nginx-devel/2013-November/004523.html > >> > > >> > -- > >> > Maxim Dounin > >> > http://nginx.org/ > >> > > >> > _______________________________________________ > >> > nginx-devel mailing list > >> > nginx-devel at nginx.org > >> > http://mailman.nginx.org/mailman/listinfo/nginx-devel > >> > >> _______________________________________________ > >> nginx-devel mailing list > >> nginx-devel at nginx.org > >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > > -- > > Maxim Dounin > > http://nginx.org/ > > > > _______________________________________________ > > nginx-devel mailing list > > nginx-devel at nginx.org > > http://mailman.nginx.org/mailman/listinfo/nginx-devel > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -- [image: HootSuite] *Adam Arsenault* Senior Software Engineer, Mobile Web | HootSuite @Adam_Arsenault | hootsuite | blog | facebook -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdasilvayy at gmail.com Tue Apr 29 20:29:29 2014 From: fdasilvayy at gmail.com (Filipe Da Silva) Date: Tue, 29 Apr 2014 22:29:29 +0200 Subject: [Patch] OCSP stapling: missing OCSP request free call Message-ID: Hello, I think there is a free call to do when the OCSP request is well created, and encoded. Or did I miss something ? Regards, Filipe # HG changeset patch # User Filipe da Silva # Date 1398802958 -7200 # Tue Apr 29 22:22:38 2014 +0200 # Node ID 8b75fcb0055f448cd5fc4f922b5b7457b38754c4 # Parent 3a48775f1535fe37cd9c034d92c5a5e9ae208c1e OCSP stapling: missing OCSP request free call diff -r 3a48775f1535 -r 8b75fcb0055f src/event/ngx_event_openssl_stapling.c --- a/src/event/ngx_event_openssl_stapling.c Tue Apr 29 12:28:41 2014 +0400 +++ b/src/event/ngx_event_openssl_stapling.c Tue Apr 29 22:22:38 2014 +0200 @@ -1195,6 +1195,8 @@ ngx_ssl_ocsp_create_request(ngx_ssl_ocsp b->last = p; ctx->request = b; + OCSP_REQUEST_free(ocsp); + return NGX_OK; failed: From mdounin at mdounin.ru Wed Apr 30 15:18:14 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 30 Apr 2014 15:18:14 +0000 Subject: [nginx] Configure: typo fixed. Message-ID: details: http://hg.nginx.org/nginx/rev/0e1491139947 branches: changeset: 5678:0e1491139947 user: Maxim Dounin date: Wed Apr 30 19:16:30 2014 +0400 description: Configure: typo fixed. diffstat: auto/lib/google-perftools/conf | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff --git a/auto/lib/google-perftools/conf b/auto/lib/google-perftools/conf --- a/auto/lib/google-perftools/conf +++ b/auto/lib/google-perftools/conf @@ -52,7 +52,7 @@ else cat << END -$0: error: the Google perftool module requires the Google perftools +$0: error: the Google perftools module requires the Google perftools library. You can either do not enable the module or install the library. END From mdounin at mdounin.ru Wed Apr 30 15:18:15 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 30 Apr 2014 15:18:15 +0000 Subject: [nginx] Cache: added ngx_quit check to ngx_http_file_cache_expir... Message-ID: details: http://hg.nginx.org/nginx/rev/5748ce74c1cc branches: changeset: 5679:5748ce74c1cc user: Maxim Dounin date: Wed Apr 30 19:16:35 2014 +0400 description: Cache: added ngx_quit check to ngx_http_file_cache_expire(). While managing big caches it is possible that expiring old cache items in ngx_http_file_cache_expire() will take a while. Added a check for ngx_quit / ngx_terminate to make sure cache manager can be terminated while in ngx_http_file_cache_expire(). diffstat: src/http/ngx_http_file_cache.c | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diffs (15 lines): diff --git a/src/http/ngx_http_file_cache.c b/src/http/ngx_http_file_cache.c --- a/src/http/ngx_http_file_cache.c +++ b/src/http/ngx_http_file_cache.c @@ -1313,6 +1313,11 @@ ngx_http_file_cache_expire(ngx_http_file for ( ;; ) { + if (ngx_quit || ngx_terminate) { + wait = 1; + break; + } + if (ngx_queue_empty(&cache->sh->queue)) { wait = 10; break; From mdounin at mdounin.ru Wed Apr 30 15:18:17 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 30 Apr 2014 15:18:17 +0000 Subject: [nginx] Core: fixed error handling in ngx_reopen_files(). Message-ID: details: http://hg.nginx.org/nginx/rev/dfb3c15bc851 branches: changeset: 5680:dfb3c15bc851 user: Maxim Dounin date: Wed Apr 30 19:16:40 2014 +0400 description: Core: fixed error handling in ngx_reopen_files(). Found by Coverity (CID 1087509). diffstat: src/core/ngx_cycle.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diffs (30 lines): diff --git a/src/core/ngx_cycle.c b/src/core/ngx_cycle.c --- a/src/core/ngx_cycle.c +++ b/src/core/ngx_cycle.c @@ -1104,6 +1104,8 @@ ngx_reopen_files(ngx_cycle_t *cycle, ngx ngx_close_file_n " \"%s\" failed", file[i].name.data); } + + continue; } if (fi.st_uid != user) { @@ -1117,6 +1119,8 @@ ngx_reopen_files(ngx_cycle_t *cycle, ngx ngx_close_file_n " \"%s\" failed", file[i].name.data); } + + continue; } } @@ -1133,6 +1137,8 @@ ngx_reopen_files(ngx_cycle_t *cycle, ngx ngx_close_file_n " \"%s\" failed", file[i].name.data); } + + continue; } } } From mdounin at mdounin.ru Wed Apr 30 15:18:18 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 30 Apr 2014 15:18:18 +0000 Subject: [nginx] Core: improved ngx_conf_parse() error handling. Message-ID: details: http://hg.nginx.org/nginx/rev/56ad171c3dd1 branches: changeset: 5681:56ad171c3dd1 user: Maxim Dounin date: Wed Apr 30 19:16:49 2014 +0400 description: Core: improved ngx_conf_parse() error handling. Previous code failed to properly restore cf->conf_file in case of ngx_close_file() errors, potentially resulting in double free of cf->conf_file->buffer->start. Found by Coverity (CID 1087507). diffstat: src/core/ngx_conf_file.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff --git a/src/core/ngx_conf_file.c b/src/core/ngx_conf_file.c --- a/src/core/ngx_conf_file.c +++ b/src/core/ngx_conf_file.c @@ -266,7 +266,7 @@ done: ngx_log_error(NGX_LOG_ALERT, cf->log, ngx_errno, ngx_close_file_n " %s failed", filename->data); - return NGX_CONF_ERROR; + rc = NGX_ERROR; } cf->conf_file = prev; From mdounin at mdounin.ru Wed Apr 30 15:18:20 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 30 Apr 2014 15:18:20 +0000 Subject: [nginx] Upstream: restored workaround for "if". Message-ID: details: http://hg.nginx.org/nginx/rev/110b50657d77 branches: changeset: 5682:110b50657d77 user: Maxim Dounin date: Wed Apr 30 19:16:55 2014 +0400 description: Upstream: restored workaround for "if". The 7022564a9e0e changeset made ineffective workaround from 2464ccebdb52 to avoid NULL pointer dereference with "if". It is now restored by moving the u->ssl_name initialization after the check. Found by Coverity (CID 1210408). diffstat: src/http/ngx_http_upstream.c | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diffs (24 lines): diff --git a/src/http/ngx_http_upstream.c b/src/http/ngx_http_upstream.c --- a/src/http/ngx_http_upstream.c +++ b/src/http/ngx_http_upstream.c @@ -584,9 +584,6 @@ ngx_http_upstream_init_request(ngx_http_ if (u->resolved == NULL) { uscf = u->conf->upstream; -#if (NGX_HTTP_SSL) - u->ssl_name = uscf->host; -#endif } else { @@ -680,6 +677,10 @@ found: return; } +#if (NGX_HTTP_SSL) + u->ssl_name = uscf->host; +#endif + if (uscf->peer.init(r, uscf) != NGX_OK) { ngx_http_upstream_finalize_request(r, u, NGX_HTTP_INTERNAL_SERVER_ERROR); From mdounin at mdounin.ru Wed Apr 30 17:00:13 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 30 Apr 2014 17:00:13 +0000 Subject: [nginx] OCSP stapling: missing OCSP request free call. Message-ID: details: http://hg.nginx.org/nginx/rev/48c97d83ab7f branches: changeset: 5683:48c97d83ab7f user: Filipe da Silva date: Tue Apr 29 22:22:38 2014 +0200 description: OCSP stapling: missing OCSP request free call. diffstat: src/event/ngx_event_openssl_stapling.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diffs (12 lines): diff --git a/src/event/ngx_event_openssl_stapling.c b/src/event/ngx_event_openssl_stapling.c --- a/src/event/ngx_event_openssl_stapling.c +++ b/src/event/ngx_event_openssl_stapling.c @@ -1195,6 +1195,8 @@ ngx_ssl_ocsp_create_request(ngx_ssl_ocsp b->last = p; ctx->request = b; + OCSP_REQUEST_free(ocsp); + return NGX_OK; failed: From mdounin at mdounin.ru Wed Apr 30 17:00:40 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 30 Apr 2014 21:00:40 +0400 Subject: [Patch] OCSP stapling: missing OCSP request free call In-Reply-To: References: Message-ID: <20140430170040.GP34696@mdounin.ru> Hello! On Tue, Apr 29, 2014 at 10:29:29PM +0200, Filipe Da Silva wrote: > Hello, > > I think there is a free call to do when the OCSP request is well > created, and encoded. > Or did I miss something ? > > Regards, > Filipe > > > # HG changeset patch > # User Filipe da Silva > # Date 1398802958 -7200 > # Tue Apr 29 22:22:38 2014 +0200 > # Node ID 8b75fcb0055f448cd5fc4f922b5b7457b38754c4 > # Parent 3a48775f1535fe37cd9c034d92c5a5e9ae208c1e > OCSP stapling: missing OCSP request free call > > diff -r 3a48775f1535 -r 8b75fcb0055f src/event/ngx_event_openssl_stapling.c > --- a/src/event/ngx_event_openssl_stapling.c Tue Apr 29 12:28:41 2014 +0400 > +++ b/src/event/ngx_event_openssl_stapling.c Tue Apr 29 22:22:38 2014 +0200 > @@ -1195,6 +1195,8 @@ ngx_ssl_ocsp_create_request(ngx_ssl_ocsp > b->last = p; > ctx->request = b; > > + OCSP_REQUEST_free(ocsp); > + > return NGX_OK; > > failed: Committed, thanks! -- Maxim Dounin http://nginx.org/ From yu.zhao at getcwd.com Wed Apr 30 21:08:57 2014 From: yu.zhao at getcwd.com (Yu Zhao) Date: Wed, 30 Apr 2014 14:08:57 -0700 Subject: nginx/backend keepalive question Message-ID: Greetings! I couldn't find my answer of following questions in the manual or somewhere else. Your help is much appreciated! 1) does nginx manage frontend/nginx and nginx/backend connections separately? In another word, will a nginx/backend connection be bond to a certain frontend/nginx connection for the lifetime of the frontend/nginx connection that is also kept-alive? My guess is it's not, nginx uses M:N model which M is the number of frontend/nginx connections that may or may not be kept-alive and N is the number of nginx/backend kept-alive connections, and M connections share N connections. 2) If my guess is correct, how does nginx make sure the request/response pair is not broken/mismatched? Nginx wil NOT send new request to a backend until the response of the last one is received, right? 3) If I'm still on the right track: because nginx has to wait the response from a backend connection, it cannot reuse it for next request. That's why nginx still needs to open new connections to backend even there are existing connections. But It will only do so when all the existing connection are non-idle (i.e. responses from them haven't been fully received). Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: