nginx-devel May 2011

nginx-devel@nginx.org

20 participants
21 discussions

by Neil Mckee

Hello, I have written a module to implement sFlow in nginx (nginx-sflow-module.googlecode.com). I'm simulating a 1-second timer-tick by assuming that the request handler will be called at least once per second. That's probably a safe assumption for any server that would care about sFlow monitoring, but I expect there's a better way... I tried asking for a timer callback like this: ngx_event_t *ev = ngx_pcalloc(pool, sizeof(ngx_event_t)); ev->hander = ngx_http_sflow_tick_event_hander; ngx_add_timer(ev, 1000); but (like most russian girls) the event never called me back. It looks like I might have to hang this on a file-descriptor somehow, but that's where I'm getting lost. Any pointers would be most appreciated. Neil

8 years, 11 months

totally transparent proxying with nginx on openbsd

by David Gwynne

openbsd has a setsockopt option called SO_BINDANY that allows a process to bind to any ip address, even if it is not local to the system. the patch below uses it to allow nginx to connect to a backend server using the ip of the client making the request. my main goal here is to allow the backend server to know the ip of the client actually making the request without having to look at extra hhtp headers i thought id throw this out there to get some help since this is my first attempt at tweaking nginx. there are a few issues with this implementation: 1. it is completely specific to openbsd. 2. it needs root privileges to use the SO_BINDANY sockopt. 3. im not sure if connections to backends are cached. if so then it is probable that a different client will reuse a previous clients proxy connection, so it will appear that the same client made both requests to the backend. to use this you just configure nginx to run as root and add "proxy_transparent on" to the sections you want this feature enabled on. you will need to add appropriate "pass out proto tcp divert-reply" rules to pf for the SO_BINDANY sockopt to work too. if anyone has some tips on how to handle problems 2 and 3 i would be grateful. cheers, dlg --- src/event/ngx_event_connect.c.orig Thu Nov 26 04:03:59 2009 +++ src/event/ngx_event_connect.c Thu Oct 28 23:22:37 2010 @@ -11,7 +11,7 @@ ngx_int_t -ngx_event_connect_peer(ngx_peer_connection_t *pc) +ngx_event_connect_peer(ngx_peer_connection_t *pc, ngx_connection_t *cc) { int rc; ngx_int_t event; @@ -20,6 +20,7 @@ ngx_event_connect_peer(ngx_peer_connection_t *pc) ngx_socket_t s; ngx_event_t *rev, *wev; ngx_connection_t *c; + int bindany; rc = pc->get(pc, pc->data); if (rc != NGX_OK) { @@ -46,6 +47,40 @@ ngx_event_connect_peer(ngx_peer_connection_t *pc) } return NGX_ERROR; + } + + if (cc != NULL) { + bindany = 1; + if (setsockopt(s, SOL_SOCKET, SO_BINDANY, + &bindany, sizeof(bindany)) == -1) + { + ngx_log_error(NGX_LOG_ALERT, pc->log, ngx_socket_errno, + "setsockopt(SO_BINDANY) failed"); + + ngx_free_connection(c); + + if (ngx_close_socket(s) == -1) { + ngx_log_error(NGX_LOG_ALERT, pc->log, ngx_socket_errno, + ngx_close_socket_n " failed"); + } + + return NGX_ERROR; + } + + if (bind(s, cc->sockaddr, cc->socklen) == -1) + { + ngx_log_error(NGX_LOG_ALERT, pc->log, ngx_socket_errno, + "bind() failed"); + + ngx_free_connection(c); + + if (ngx_close_socket(s) == -1) { + ngx_log_error(NGX_LOG_ALERT, pc->log, ngx_socket_errno, + ngx_close_socket_n " failed"); + } + + return NGX_ERROR; + } } if (pc->rcvbuf) { --- src/event/ngx_event_connect.h.orig Tue Nov 3 01:24:02 2009 +++ src/event/ngx_event_connect.h Thu Oct 28 23:22:37 2010 @@ -68,7 +68,8 @@ struct ngx_peer_connection_s { }; -ngx_int_t ngx_event_connect_peer(ngx_peer_connection_t *pc); +ngx_int_t ngx_event_connect_peer(ngx_peer_connection_t *pc, + ngx_connection_t *cc); ngx_int_t ngx_event_get_peer(ngx_peer_connection_t *pc, void *data); --- src/http/modules/ngx_http_proxy_module.c.orig Mon May 24 21:01:05 2010 +++ src/http/modules/ngx_http_proxy_module.c Thu Oct 28 23:42:10 2010 @@ -71,6 +71,7 @@ typedef struct { ngx_http_proxy_vars_t vars; ngx_flag_t redirect; + ngx_flag_t transparent; ngx_uint_t headers_hash_max_size; ngx_uint_t headers_hash_bucket_size; @@ -196,6 +197,13 @@ static ngx_command_t ngx_http_proxy_commands[] = { 0, NULL }, + { ngx_string("proxy_transparent"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_FLAG, + ngx_conf_set_flag_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_proxy_loc_conf_t, transparent), + NULL }, + { ngx_string("proxy_store"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, ngx_http_proxy_store, @@ -626,6 +634,7 @@ ngx_http_proxy_handler(ngx_http_request_t *r) u->abort_request = ngx_http_proxy_abort_request; u->finalize_request = ngx_http_proxy_finalize_request; r->state = 0; + r->transparent = (plcf->transparent == 1); if (plcf->redirects) { u->rewrite_redirect = ngx_http_proxy_rewrite_redirect; @@ -1940,6 +1949,7 @@ ngx_http_proxy_create_loc_conf(ngx_conf_t *cf) conf->upstream.cyclic_temp_file = 0; conf->redirect = NGX_CONF_UNSET; + conf->transparent = NGX_CONF_UNSET; conf->upstream.change_buffering = 1; conf->headers_hash_max_size = NGX_CONF_UNSET_UINT; @@ -2214,6 +2224,8 @@ ngx_http_proxy_merge_loc_conf(ngx_conf_t *cf, void *pa } } } + + ngx_conf_merge_value(conf->transparent, prev->transparent, 0); /* STUB */ if (prev->proxy_lengths) { --- src/http/ngx_http_request.h.orig Mon May 24 22:35:10 2010 +++ src/http/ngx_http_request.h Thu Oct 28 23:22:37 2010 @@ -511,6 +511,8 @@ struct ngx_http_request_s { unsigned stat_writing:1; #endif + unsigned transparent:1; + /* used to parse HTTP headers */ ngx_uint_t state; --- src/http/ngx_http_upstream.c.orig Mon May 24 22:35:10 2010 +++ src/http/ngx_http_upstream.c Thu Oct 28 23:22:37 2010 @@ -1066,7 +1066,8 @@ ngx_http_upstream_connect(ngx_http_request_t *r, ngx_h u->state->response_sec = tp->sec; u->state->response_msec = tp->msec; - rc = ngx_event_connect_peer(&u->peer); + rc = ngx_event_connect_peer(&u->peer, r->transparent ? + r->connection : NULL); ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "http upstream connect: %i", rc); --- src/mail/ngx_mail_auth_http_module.c.orig Fri May 14 19:56:37 2010 +++ src/mail/ngx_mail_auth_http_module.c Thu Oct 28 23:22:37 2010 @@ -191,7 +191,7 @@ ngx_mail_auth_http_init(ngx_mail_session_t *s) ctx->peer.log = s->connection->log; ctx->peer.log_error = NGX_ERROR_ERR; - rc = ngx_event_connect_peer(&ctx->peer); + rc = ngx_event_connect_peer(&ctx->peer, NULL); if (rc == NGX_ERROR || rc == NGX_BUSY || rc == NGX_DECLINED) { if (ctx->peer.connection) { --- src/mail/ngx_mail_proxy_module.c.orig Thu Oct 28 23:32:15 2010 +++ src/mail/ngx_mail_proxy_module.c Thu Oct 28 23:30:53 2010 @@ -147,7 +147,7 @@ ngx_mail_proxy_init(ngx_mail_session_t *s, ngx_addr_t p->upstream.log = s->connection->log; p->upstream.log_error = NGX_ERROR_ERR; - rc = ngx_event_connect_peer(&p->upstream); + rc = ngx_event_connect_peer(&p->upstream, NULL); if (rc == NGX_ERROR || rc == NGX_BUSY || rc == NGX_DECLINED) { ngx_mail_proxy_internal_server_error(s);

9 years, 7 months

[PATCH] Fix ./configure script for less error-forgiving compilers

by Piotr Sikora

Hi Igor, attached patch fixes ./configure script for less error-forgiving compilers (like LLVM/clang) when they are used with -Werror option. auto/types/sizeof: - printf() is defined in missing <stdio.h> header, - sizeof() returns size_t type, which has different signedness on different operating systems (hence %z). auto/unix: - setproctitle() is defined in <stdlib.h> on both: NetBSD and OpenBSD. Best regards, Piotr Sikora < piotr.sikora(a)frickle.com >

10 years, 5 months

Patch for Age header

by Woon Wai Keen

Hi all, I would like to contribute a patch to provide the cached file age as a variable $proxy_cache_age , which can then be used together with add_header Age $proxy_cache_age. The age calculation is not RFC2616 compliant (13.2.3) as it doesn't take into consideration the cached origin Age header and Date. Is it feasible to extract this from the cached headers? -- regards, wK

10 years, 10 months

[PATCH] Fastcgi: core dump was caused by duplicated request header

by Simon Liu

this bug will give rise to nginx(version >= 0.8.40) core dump, and it was caused by this feature: *) Feature: a "fastcgi_param" directive with value starting with "HTTP_" overrides a client request header line. When we difine fastcgi_param directive with value starting with "HTTP_", nginx malloc a array(size is header_params that is number of value starting with "HTTP_"), and if request header contain this value(HTTTP_xxx), nginx will add this header pointer to array, but if header is duplicated, this array will cross-border. e.g. if the config contain this directive (fastcgi_param HTTP_HOST $http_host), and then request header send multi-duplicated header(Host), nginx will core dump. Index: src/http/modules/ngx_http_fastcgi_module.c =================================================================== --- src/http/modules/ngx_http_fastcgi_module.c (revision 3929) +++ src/http/modules/ngx_http_fastcgi_module.c (working copy) @@ -165,7 +165,10 @@ static char *ngx_http_fastcgi_lowat_check(ngx_conf_t *cf, void *post, void *data); +static ngx_inline ngx_int_t ngx_http_fastcgi_ignored_header(ngx_uint_t hash, + ngx_uint_t header_params, ngx_uint_t *ignored); + static ngx_conf_post_t ngx_http_fastcgi_lowat_post = { ngx_http_fastcgi_lowat_check }; @@ -684,18 +687,32 @@ #endif +static ngx_inline ngx_int_t +ngx_http_fastcgi_ignored_header(ngx_uint_t hash, ngx_uint_t header_params, ngx_uint_t *ignored) +{ + ngx_uint_t n; + + for (n = 0; n < header_params; n++) { + if (hash == ignored[n]) { + return NGX_OK; + } + } + + return NGX_DECLINED; +} + + static ngx_int_t ngx_http_fastcgi_create_request(ngx_http_request_t *r) { off_t file_pos; - u_char ch, *pos, *lowcase_key; - size_t size, len, key_len, val_len, padding, - allocated; - ngx_uint_t i, n, next, hash, header_params; + u_char ch, *pos; + size_t size, len, key_len, val_len, padding; + ngx_uint_t i, n, next, header_params, *ignored; ngx_buf_t *b; ngx_chain_t *cl, *body; ngx_list_part_t *part; - ngx_table_elt_t *header, **ignored; + ngx_table_elt_t *header; ngx_http_script_code_pt code; ngx_http_script_engine_t e, le; ngx_http_fastcgi_header_t *h; @@ -733,11 +750,8 @@ if (flcf->upstream.pass_request_headers) { - allocated = 0; - lowcase_key = NULL; - if (flcf->header_params) { - ignored = ngx_palloc(r->pool, flcf->header_params * sizeof(void *)); + ignored = ngx_palloc(r->pool, flcf->header_params * sizeof(ngx_uint_t)); if (ignored == NULL) { return NGX_ERROR; } @@ -759,43 +773,21 @@ } if (flcf->header_params) { - if (allocated < header[i].key.len) { - allocated = header[i].key.len + 16; - lowcase_key = ngx_pnalloc(r->pool, allocated); - if (lowcase_key == NULL) { - return NGX_ERROR; - } - } - hash = 0; - - for (n = 0; n < header[i].key.len; n++) { - ch = header[i].key.data[n]; - - if (ch >= 'A' && ch <= 'Z') { - ch |= 0x20; - - } else if (ch == '-') { - ch = '_'; + if (ngx_hash_find(&flcf->headers_hash, header[i].hash, + header[i].lowcase_key, header[i].key.len)) { + if (ngx_http_fastcgi_ignored_header(header[i].hash, + header_params, ignored) == NGX_OK) { + continue; } - - hash = ngx_hash(hash, ch); - lowcase_key[n] = ch; - } - - if (ngx_hash_find(&flcf->headers_hash, hash, lowcase_key, n)) { - ignored[header_params++] = &header[i]; + ignored[header_params++] = header[i].hash; continue; } - - n += sizeof("HTTP_") - 1; - - } else { - n = sizeof("HTTP_") - 1 + header[i].key.len; } - - len += ((n > 127) ? 4 : 1) + ((header[i].value.len > 127) ? 4 : 1) - + n + header[i].value.len; + + len += ((sizeof("HTTP_") - 1 + header[i].key.len > 127) ? 4 : 1) + + ((header[i].value.len > 127) ? 4 : 1) + + sizeof("HTTP_") - 1 + header[i].key.len + header[i].value.len; } } @@ -915,10 +907,9 @@ i = 0; } - for (n = 0; n < header_params; n++) { - if (&header[i] == ignored[n]) { - goto next; - } + if (ngx_http_fastcgi_ignored_header(header[i].hash, + header_params, ignored) == NGX_OK) { + continue; } key_len = sizeof("HTTP_") - 1 + header[i].key.len; @@ -964,8 +955,6 @@ "fastcgi param: \"%*s: %*s\"", key_len, b->last - (key_len + val_len), val_len, b->last - val_len); - next: - continue; } } @@ -2013,10 +2002,10 @@ ngx_http_fastcgi_loc_conf_t *prev = parent; ngx_http_fastcgi_loc_conf_t *conf = child; - u_char *p; + u_char ch, *p; size_t size; uintptr_t *code; - ngx_uint_t i; + ngx_uint_t i, n, params_hash; ngx_array_t headers_names; ngx_keyval_t *src; ngx_hash_key_t *hk; @@ -2374,7 +2363,22 @@ hk->key.len = src[i].key.len - 5; hk->key.data = src[i].key.data + 5; - hk->key_hash = ngx_hash_key_lc(hk->key.data, hk->key.len); + + params_hash = 0; + for (n = 5; n < src[i].key.len; n++) { + ch = src[i].key.data[n]; + + if (ch >= 'A' && ch <= 'Z') { + ch |= 0x20; + + } else if (ch == '_') { + ch = '-'; + } + + params_hash = ngx_hash(params_hash, ch); + } + + hk->key_hash = params_hash; hk->value = (void *) 1; if (src[i].value.len == 0) { -- 博观约取豆瓣:www.douban.com/people/mustang/ blog: www.pagefault.info twitter: www.twitter.com/minibobo sina 微博: www.weibo.com/diaoliang

11 years, 2 months

Content does not get replaced

by Adelino Monteiro

Hello, I'm writing my first module (actually a copy from mod_strip with some simple modifications) I have this simple function that for testing purposes should simply make a simple substitution of all chars to A. I used gdb and chain_link->buf->start indeed has all the characters replaced with A. However the output in the end is the original file without any other modification. Could someone shed some light on this mistery? Thanks AM static ngx_int_t ngx_http_strip_body_filter(ngx_http_request_t *r, ngx_chain_t *in) { ngx_http_strip_ctx_t *ctx; ngx_chain_t *chain_link; u_char *reader; u_char *writer; ngx_log_t *log; log = r->connection->log; ngx_log_debug0(NGX_LOG_DEBUG_HTTP, log, 0, "In strip strip body filter 1 "); ctx = ngx_http_get_module_ctx(r, ngx_http_strip_filter_module); if (ctx == NULL) { ngx_log_debug0(NGX_LOG_DEBUG_HTTP, log, 0, "In strip strip body filter 2 "); return ngx_http_next_body_filter(r, in); } ngx_log_debug0(NGX_LOG_DEBUG_HTTP, log, 0, "In strip strip body filter 3 "); for (chain_link = in; chain_link; chain_link = chain_link->next) { for (writer = chain_link->buf->pos, reader = chain_link->buf->pos; reader < chain_link->buf->last; reader++) { ngx_log_debug1(NGX_LOG_DEBUG_HTTP, log, 0, "w: \"%c\"", *reader); *reader = 'A' ; if (reader < chain_link->buf->last) *writer++ = *reader; } ngx_log_debug1(NGX_LOG_DEBUG_HTTP, log, 0, "In strip_body_filter writer: \"%s\"", writer); chain_link->buf->last = writer; } return ngx_http_next_body_filter(r, in); }

11 years, 2 months

ngx_http_read_client_request_body() causing random timeouts

by Srebrenko Šehić

Hi, I've been trying to identify what is causing timeouts towards the client when a POST request is sent to nginx. I have a custom module that reads the POST payload and runs in ACCESS phase. It's based on nginx-form-input module from agentz. Normally, the modules works OK, but it sometimes times out and nginx logs the following: 2011/05/31 09:31:16 [alert] 25731#0: *1 zero size buf in output t:1 r:0 f:0 0000000204DFD000 0000000204DFD000-0000000204DFD000 0000000000000000 0-0 while sending request to upstream, client: 192.168.0.27, server: 192.168.0.27, request: "POST /slurp_md5.php HTTP/1.1", upstream: "http://192.168.0.9:80/slurp_md5.php", host: "192.168.0.27" I've noticed that this only happens with keep-alive connections. Can anybody shed some light on this? I've looked at other modules and nginx core, but can't seem to figure what could be causing this. Thanks. Code for the sample module is enclosed below. nginx is vanilla 1.0.3 (with perl module compiled). Cheers, Srebrenko ---- module source code ---- #include <ngx_config.h> #include <ngx_core.h> #include <ngx_http.h> static ngx_int_t ngx_http_dummy_post_read_init(ngx_conf_t *cf); static ngx_int_t ngx_http_dummy_post_read_access_handler(ngx_http_request_t *r); typedef struct { ngx_flag_t done:1; ngx_flag_t waiting_more_body:1; } ngx_http_dummy_post_read_ctx_t; static ngx_http_module_t ngx_http_dummy_post_read_module_ctx = { NULL, /* preconfiguration */ ngx_http_dummy_post_read_init, /* postconfiguration */ NULL, /* create main configuration */ NULL, /* init main configuration */ NULL, /* create server configuration */ NULL, /* merge server configuration */ NULL, /* create location configuration */ NULL /* merge location configuration */ }; ngx_module_t ngx_http_dummy_post_read_module = { NGX_MODULE_V1, &ngx_http_dummy_post_read_module_ctx, /* module context */ NULL, /* module directives */ NGX_HTTP_MODULE, /* module type */ NULL, /* init master */ NULL, /* init module */ NULL, /* init process */ NULL, /* init thread */ NULL, /* exit thread */ NULL, /* exit process */ NULL, /* exit master */ NGX_MODULE_V1_PADDING }; void ngx_http_dummy_post_read_payload_handler(ngx_http_request_t *r) { ngx_http_dummy_post_read_ctx_t *ctx; r->read_event_handler = ngx_http_request_empty_handler; ctx = ngx_http_get_module_ctx(r, ngx_http_dummy_post_read_module); ctx->done = 1; r->main->count--; if (ctx->waiting_more_body) { ctx->waiting_more_body = 0; ngx_http_core_run_phases(r); } } static ngx_int_t ngx_http_dummy_post_read_access_handler(ngx_http_request_t *r) { ngx_int_t rc; ngx_http_dummy_post_read_ctx_t *ctx; ctx = ngx_http_get_module_ctx(r, ngx_http_dummy_post_read_module); if (ctx != NULL) { if (ctx->done) { return NGX_DECLINED; } return NGX_DONE; } if (r->method != NGX_HTTP_POST) { return NGX_DECLINED; } ctx = ngx_palloc(r->pool, sizeof(ngx_http_dummy_post_read_ctx_t)); if (ctx == NULL) { ngx_http_finalize_request(r, NGX_ERROR); return NGX_ERROR; } ngx_http_set_ctx(r, ctx, ngx_http_dummy_post_read_module); rc = ngx_http_read_client_request_body(r, ngx_http_dummy_post_read_payload_handler); if (rc == NGX_ERROR || rc >= NGX_HTTP_SPECIAL_RESPONSE) { return rc; } if (rc == NGX_AGAIN) { ctx->waiting_more_body = 1; return NGX_DONE; } return NGX_DECLINED; } static ngx_int_t ngx_http_dummy_post_read_init(ngx_conf_t *cf) { ngx_http_handler_pt *h; ngx_http_core_main_conf_t *cmcf; cmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_core_module); h = ngx_array_push(&cmcf->phases[NGX_HTTP_ACCESS_PHASE].handlers); if (h == NULL) { return NGX_ERROR; } *h = ngx_http_dummy_post_read_access_handler; return NGX_OK; }

11 years, 2 months

Our nginx module development roadmap (draft)

by agentzh

Hi, folks! Here's the first draft of our nginx module development plan in the next year or so: http://agentzh.org/misc/nginx/roadmap.html We'll keep this roadmap document up to date. Comments and suggestions will be highly appreciated as usual :) Cheers, -agentzh

11 years, 2 months

Is there a way to hook into between accept() and ssl calls?

by Qun-Ying

Hi, Is there a way to hook into the state in between accept() and before doing any ssl stuff? -- Qun-Ying

11 years, 2 months

{module dev}include <sys/time.h> on 32bit linux cause segmentation fault?

by Conan

hi, I'm wring a nginx module which is hooked at NGX_HTTP_ACCESS_PHASE. Segmentation fault happened when calling ngx_log_error(NGX_LOG_ERR, r->connection->log, "xxx") in my module's handler(which will handle every request). I found r->connection->log = 0x0 when debug with gdb. On 64bit centos/ubuntu/macos module works well. Segmentation fault seem only occurs on 32bit linux(ubuntu). When I comment out #include <sys/time.h>, segmentation fault disappeared. Attachment is my module's skeleton.

11 years, 2 months

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

nginx-devel May 2011