[PATCH] Mail: added PROXY PROTOCOL support

Maxim Dounin mdounin at mdounin.ru
Sat Feb 20 13:44:57 UTC 2021


Hello!

On Tue, Jan 19, 2021 at 06:34:30PM +0300, muradm wrote:

> # HG changeset patch
> # User muradm <mail at muradm.net>
> # Date 1611069863 -10800
> #      Tue Jan 19 18:24:23 2021 +0300
> # Node ID 4618e767b84c5b3a7712466edb5bf37e3f0294ed
> # Parent  83c4622053b02821a12d522d08eaff3ac27e65e3

Thanks for the patch.  Overall it looks much better than previous 
attempts and close to what I would like to see implemented.  See 
below for additional comments.

> Mail: added PROXY PROTOCOL support.

The specification spells it as "PROXY protocol"
(https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt).

> 
> This implements propxy protocol support for both upstream and downstream.
> 
> Downstream proxy protocol support:
> 
> mail {
>     server {
>         listen <port> [ssl] proxy_protocol;
> 	protocol <imap|pop3|smtp>;

Just a side notee: "[ssl]" in the examples looks unneeded.  There 
are multiple possible parameters of the listen directive, and 
listing just one of them doesn't look right.  On the other hand, 
providing full current syntax of the listen directive would be a 
bad idea as well.

The same applies to the "protocol <...>;" directive.  This 
directive isn't needed when configuring mail proxy servers on 
standard ports.  If it's needed, this is something up to the user 
to configure.

>     }
> }
> 
> This will properly handle incoming connections from load balancer sending
> PROXY protocol header. Without this, it is impossible to run nginx mail
> proxy behind such balancer. Header reading is done with existing function
> "ngx_proxy_protocol_read", so it should support both v1 and v2 headers.
> This will also set "sockaddr" and "local_sockaddr" addresses from received
> header, mimicing "set_realip". While "realip_module" deals with variables
> etc., which is necessary for HTTP protocol, mail protocols are pretty
> strict, so there is no need for flexible handling of real addresses
> received.
> 
> Upstream proxy protocol support:
> 
> mail {
>     server {
>         listen <port> [ssl];
> 	protocol <imap|pop3|smtp>;
> 	proxy_protocol on;
>     }
> }
> 
> With this, upstream server (like Postfix, Exim, Dovecot) will have PROXY
> protocol header. Mentioned programs do support proxy protocol out of the
> box. Header is written with existing function "ngx_proxy_protocol_write"
> which supports only v1 header writing. Contents of header are written
> from "sockaddr" and "local_sockaddr".

These are two distinct features: reading PROXY protocol header 
provided by the balancer, and sending PROXY protocol to backends.  
As such, these should be in separate patches.  In particular, this 
will simplify review.

> 
> Downstream and upstream proxy protocol support:
> 
> mail {
>     server {
>         listen <port> [ssl] proxy_protocol;
> 	protocol <imap|pop3|smtp>;
> 	proxy_protocol on;
>     }
> }
> 
> This will combine both receiving PROXY header and sending PROXY header. With
> this, upstream server (like Postfix, Exim, Dovecot) will receive the same
> header as was sent by downstream load balancer.
> 
> Above configurations work for SSL as well and should be transparent to other
> mail related configurations.
> 
> Added upstream server "connect_timeout" which defaults to 1 second.

There is no need to introduce additional timeouts here: there is 
already the "timeout" directive (cscf->timeout), which limits 
aggregate timeout for the full login process, including connect.

If you think that a separate connect_timeout is needed for 
whatever reason, consider submitting a separate patch - with 
appropriate reasoning.

> 
> Server configurations enabling proxy_protocol in listen directive, require
> "set_real_ip_from" configuration. Like the following:
> 
> mail {
>     # ...
>     server {
>         listen 587 proxy_protocol;
> 	set_real_ip_from "192.168.1.1";
> 	set_real_ip_from "10.10.0.0/16";
> 	set_real_ip_from "0.0.0.0/0";
>     }
> }
> 
> With enabled "proxy_protocol" and missing at least one "set_real_ip_from",
> all connections will be dropped and at startup user will see in error_log:
> 
>     using PROXY protocol without set_real_ip_from \
>         while reading PROXY protocol header
> 
> When "set_real_ip_from" is provided, but remote address on physical connection
> does not satisfy any address criteria, at "notice" level, in error_log, user
> will see:
> 
>     UNTRUSTED PROXY protocol provider: 127.0.0.1 \
>         while reading PROXY protocol header, \
> 	client: 127.0.0.1, server: 127.0.0.1:8143

Such behaviour, "reject connection if not matched by 
set_real_ip_from", looks wrong.  It is not how things are handled 
in stream/http, so such behaviour will break POLA.  It also
contradicts the name of the set_real_ip_from directive: it is 
expected to set addreses got from the IPs in question, but not 
reject anything else.

Further, accepting PROXY protocol header and _not_ changing the 
address nginx thinks the connection is from looks like a valid 
use case to me.  For example, the address from PROXY protocol can 
be provided to the auth_http script, making it possible for the 
script to do appropriate decisions.

Overall, it looks like there should be at least 3 patches here: 
first one to introduce "listen ... proxy_protocol" and sending 
appropriate information to auth_http script, second one for 
"set_real_ip_from", and third one for "proxy_protocol on;" in the 
proxy module.

> 
> diff -r 83c4622053b0 -r 4618e767b84c src/mail/ngx_mail.c
> --- a/src/mail/ngx_mail.c	Tue Jan 12 16:59:31 2021 +0300
> +++ b/src/mail/ngx_mail.c	Tue Jan 19 18:24:23 2021 +0300
> @@ -402,6 +402,7 @@
>          addrs[i].addr = sin->sin_addr.s_addr;
>  
>          addrs[i].conf.ctx = addr[i].opt.ctx;
> +        addrs[i].conf.proxy_protocol = addr[i].opt.proxy_protocol;
>  #if (NGX_MAIL_SSL)
>          addrs[i].conf.ssl = addr[i].opt.ssl;
>  #endif
> @@ -436,6 +437,7 @@
>          addrs6[i].addr6 = sin6->sin6_addr;
>  
>          addrs6[i].conf.ctx = addr[i].opt.ctx;
> +        addrs6[i].conf.proxy_protocol = addr[i].opt.proxy_protocol;
>  #if (NGX_MAIL_SSL)
>          addrs6[i].conf.ssl = addr[i].opt.ssl;
>  #endif
> diff -r 83c4622053b0 -r 4618e767b84c src/mail/ngx_mail.h
> --- a/src/mail/ngx_mail.h	Tue Jan 12 16:59:31 2021 +0300
> +++ b/src/mail/ngx_mail.h	Tue Jan 19 18:24:23 2021 +0300
> @@ -37,6 +37,7 @@
>      unsigned                bind:1;
>      unsigned                wildcard:1;
>      unsigned                ssl:1;
> +    unsigned                proxy_protocol:1;
>  #if (NGX_HAVE_INET6)
>      unsigned                ipv6only:1;
>  #endif

It's a better idea to keep proxy_protocol similar to stream/http, 
after the so_keepalive.

> @@ -56,6 +57,7 @@
>      ngx_mail_conf_ctx_t    *ctx;
>      ngx_str_t               addr_text;
>      ngx_uint_t              ssl;    /* unsigned   ssl:1; */
> +    unsigned                proxy_protocol:1;

The "ssl" flag needs to be converted to bitfield (as specified in 
the comment) as long as another bitfield is added.

>  } ngx_mail_addr_conf_t;
>  
>  typedef struct {
> @@ -125,6 +127,8 @@
>      ngx_mail_conf_ctx_t    *ctx;
>  
>      ngx_uint_t              listen;  /* unsigned  listen:1; */
> +
> +    ngx_array_t            *realip_from;     /* array of ngx_cidr_t */
>  } ngx_mail_core_srv_conf_t;
>  
>  

It should be a good idea to put the realip_from field close to 
other configuration field of the mail core module, that is, 
somewhere before the pointer to other module configuration 
contexts.

Alternatively, a separate module for set_real_ip_from with its own 
configuration might be a good idea as well.

> @@ -190,6 +194,7 @@
>      void                  **ctx;
>      void                  **main_conf;
>      void                  **srv_conf;
> +    ngx_mail_addr_conf_t   *addr_conf;
>  
>      ngx_resolver_ctx_t     *resolver_ctx;
>  
> @@ -197,6 +202,7 @@
>  
>      ngx_uint_t              mail_state;
>  
> +    unsigned                proxy_protocol:1;
>      unsigned                protocol:3;
>      unsigned                blocked:1;
>      unsigned                quit:1;
> diff -r 83c4622053b0 -r 4618e767b84c src/mail/ngx_mail_core_module.c
> --- a/src/mail/ngx_mail_core_module.c	Tue Jan 12 16:59:31 2021 +0300
> +++ b/src/mail/ngx_mail_core_module.c	Tue Jan 19 18:24:23 2021 +0300
> @@ -25,7 +25,7 @@
>      void *conf);
>  static char *ngx_mail_core_resolver(ngx_conf_t *cf, ngx_command_t *cmd,
>      void *conf);
> -

Unrelated (and wrong from the style point of view) change, two 
empty lines between functions and the following variable should be 
preserved.

> +static char *ngx_mail_core_realip_from(ngx_conf_t *cf, ngx_command_t *cmd, void *conf);
>  
>  static ngx_command_t  ngx_mail_core_commands[] = {
>  
> @@ -85,6 +85,13 @@
>        offsetof(ngx_mail_core_srv_conf_t, resolver_timeout),
>        NULL },
>  
> +    { ngx_string("set_real_ip_from"),
> +      NGX_MAIL_MAIN_CONF|NGX_MAIL_SRV_CONF|NGX_CONF_TAKE1,
> +      ngx_mail_core_realip_from,
> +      NGX_MAIL_SRV_CONF_OFFSET,
> +      offsetof(ngx_mail_core_srv_conf_t, realip_from),
> +      NULL },
> +
>        ngx_null_command
>  };
>  
> @@ -165,6 +172,8 @@
>  
>      cscf->resolver = NGX_CONF_UNSET_PTR;
>  
> +    cscf->realip_from = NGX_CONF_UNSET_PTR;
> +
>      cscf->file_name = cf->conf_file->file.name.data;
>      cscf->line = cf->conf_file->line;
>  
> @@ -206,6 +215,10 @@
>  
>      ngx_conf_merge_ptr_value(conf->resolver, prev->resolver, NULL);
>  
> +    ngx_conf_merge_ptr_value(conf->realip_from,
> +                             prev->realip_from,
> +                             NGX_CONF_UNSET_PTR);
> +

It is generally incorrect to preserve unset values till runtime 
unless strictly necessary.  A better approach would be to follow 
how set_real_ip_from is handled in http/stream and use NULL if not 
set.  Checking for NULL to mean "no set_real_ip_from items 
defined" is also more natural.

>      return NGX_CONF_OK;
>  }
>  
> @@ -548,6 +561,11 @@
>  #endif
>          }
>  
> +        if (ngx_strcmp(value[i].data, "proxy_protocol") == 0) {
> +            ls->proxy_protocol = 1;
> +            continue;
> +        }
> +
>          ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
>                             "the invalid \"%V\" parameter", &value[i]);
>          return NGX_CONF_ERROR;
> @@ -676,3 +694,104 @@
>  
>      return NGX_CONF_OK;
>  }
> +
> +char *

Style: there should be two empty lines between functions.

Missing static specifier.

> +ngx_mail_core_realip_from(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
> +{
> +    ngx_mail_core_srv_conf_t *cscf = conf;
> +
> +    ngx_int_t             rc;
> +    ngx_str_t            *value;
> +    ngx_url_t             u;
> +    ngx_cidr_t            c, *cidr;
> +    ngx_uint_t            i;
> +    struct sockaddr_in   *sin;
> +#if (NGX_HAVE_INET6)
> +    struct sockaddr_in6  *sin6;
> +#endif
> +
> +    value = cf->args->elts;
> +
> +    if (cscf->realip_from == NGX_CONF_UNSET_PTR) {
> +        cscf->realip_from = ngx_array_create(cf->pool, 2, sizeof(ngx_cidr_t));
> +        if (cscf->realip_from == NULL) {
> +            return NGX_CONF_ERROR;
> +        }
> +    }
> +
> +#if (NGX_HAVE_UNIX_DOMAIN)
> +
> +    if (ngx_strcmp(value[1].data, "unix:") == 0) {
> +        cidr = ngx_array_push(cscf->realip_from);
> +        if (cidr == NULL) {
> +            return NGX_CONF_ERROR;
> +        }
> +
> +        cidr->family = AF_UNIX;
> +        return NGX_CONF_OK;
> +    }
> +
> +#endif
> +
> +    rc = ngx_ptocidr(&value[1], &c);
> +
> +    if (rc != NGX_ERROR) {
> +        if (rc == NGX_DONE) {
> +            ngx_conf_log_error(NGX_LOG_WARN, cf, 0,
> +                               "low address bits of %V are meaningless",
> +                               &value[1]);
> +        }
> +
> +        cidr = ngx_array_push(cscf->realip_from);
> +        if (cidr == NULL) {
> +            return NGX_CONF_ERROR;
> +        }
> +
> +        *cidr = c;
> +
> +        return NGX_CONF_OK;
> +    }
> +
> +    ngx_memzero(&u, sizeof(ngx_url_t));
> +    u.host = value[1];
> +
> +    if (ngx_inet_resolve_host(cf->pool, &u) != NGX_OK) {
> +        if (u.err) {
> +            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
> +                               "%s in set_real_ip_from \"%V\"",
> +                               u.err, &u.host);
> +        }
> +
> +        return NGX_CONF_ERROR;
> +    }
> +
> +    cidr = ngx_array_push_n(cscf->realip_from, u.naddrs);
> +    if (cidr == NULL) {
> +        return NGX_CONF_ERROR;
> +    }
> +
> +    ngx_memzero(cidr, u.naddrs * sizeof(ngx_cidr_t));
> +
> +    for (i = 0; i < u.naddrs; i++) {
> +        cidr[i].family = u.addrs[i].sockaddr->sa_family;
> +
> +        switch (cidr[i].family) {
> +
> +#if (NGX_HAVE_INET6)
> +        case AF_INET6:
> +            sin6 = (struct sockaddr_in6 *) u.addrs[i].sockaddr;
> +            cidr[i].u.in6.addr = sin6->sin6_addr;
> +            ngx_memset(cidr[i].u.in6.mask.s6_addr, 0xff, 16);
> +            break;
> +#endif
> +
> +        default: /* AF_INET */
> +            sin = (struct sockaddr_in *) u.addrs[i].sockaddr;
> +            cidr[i].u.in.addr = sin->sin_addr.s_addr;
> +            cidr[i].u.in.mask = 0xffffffff;
> +            break;
> +        }
> +    }
> +
> +    return NGX_CONF_OK;
> +}
> diff -r 83c4622053b0 -r 4618e767b84c src/mail/ngx_mail_handler.c
> --- a/src/mail/ngx_mail_handler.c	Tue Jan 12 16:59:31 2021 +0300
> +++ b/src/mail/ngx_mail_handler.c	Tue Jan 19 18:24:23 2021 +0300
> @@ -12,6 +12,8 @@
>  
>  
>  static void ngx_mail_init_session(ngx_connection_t *c);
> +static void ngx_mail_init_connection_complete(ngx_connection_t *c);
> +static void ngx_mail_proxy_protocol_handler(ngx_event_t *rev);
>  
>  #if (NGX_MAIL_SSL)
>  static void ngx_mail_ssl_init_connection(ngx_ssl_t *ssl, ngx_connection_t *c);
> @@ -128,6 +130,7 @@
>  
>      s->main_conf = addr_conf->ctx->main_conf;
>      s->srv_conf = addr_conf->ctx->srv_conf;
> +    s->addr_conf = addr_conf;

There is no need to preserve addr_conf here, relevant flags can 
(notably, ssl) can be preserved by itself, similarly to how it is 
done in ngx_stream_init_connection().

>  
>      s->addr_text = &addr_conf->addr_text;
>  
> @@ -159,13 +162,181 @@
>  
>      c->log_error = NGX_ERROR_INFO;
>  
> +    /*
> +     * Before all process proxy protocol
> +     */
> +

This looks too verbose.

> +    if (addr_conf->proxy_protocol) {
> +        s->proxy_protocol = 1;

The "s->proxy_protocol" flag looks meaningless, it is not used 
anywhere.

> +        c->log->action = "reading PROXY protocol header";
> +        c->read->handler = ngx_mail_proxy_protocol_handler;
> +
> +        ngx_add_timer(c->read, cscf->timeout);
> +
> +        if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
> +            ngx_mail_close_connection(c);
> +        }
> +
> +        return;

Here the handler is not called as long as some data is already 
available for reading.  While this is unlikely to cause problems 
right now in the mail module due to lack of accept filters / 
deferred accept support in the mail module (and iocp support being 
broken), this is generally wrong.  A better approach is to check 
c->read->ready and act appropriately, much like stream module code 
does.

> +    }
> +
> +    ngx_mail_init_connection_complete(c);
> +}
> +
> +
> +ngx_int_t
> +ngx_mail_proxy_protoco_set_addrs(ngx_connection_t *c)
> +{
> +    ngx_addr_t                addr_peer, addr_local;
> +    u_char                   *p, text[NGX_SOCKADDR_STRLEN];
> +    size_t                    len;

Style: wrong order of types, wrong amount of spaces between 
types and variable names.

> +
> +    if (ngx_parse_addr(c->pool, &addr_peer,
> +                       c->proxy_protocol->src_addr.data,
> +                       c->proxy_protocol->src_addr.len) != NGX_OK)
> +    {
> +        return NGX_ERROR;
> +    }
> +
> +    ngx_inet_set_port(addr_peer.sockaddr, c->proxy_protocol->src_port);
> +
> +    if (ngx_parse_addr(c->pool, &addr_local,
> +                       c->proxy_protocol->dst_addr.data,
> +                       c->proxy_protocol->dst_addr.len) != NGX_OK)
> +    {
> +        return NGX_ERROR;
> +    }
> +
> +    ngx_inet_set_port(addr_local.sockaddr, c->proxy_protocol->dst_port);
> +
> +    len = ngx_sock_ntop(addr_peer.sockaddr, addr_peer.socklen, text,
> +                        NGX_SOCKADDR_STRLEN, 0);
> +    if (len == 0) {
> +        return NGX_ERROR;
> +    }
> +
> +    p = ngx_pnalloc(c->pool, len);
> +    if (p == NULL) {
> +        return NGX_ERROR;
> +    }
> +
> +    ngx_memcpy(p, text, len);
> +
> +    c->sockaddr = addr_peer.sockaddr;
> +    c->socklen = addr_peer.socklen;
> +    c->addr_text.len = len;
> +    c->addr_text.data = p;
> +
> +    len = ngx_sock_ntop(addr_local.sockaddr, addr_local.socklen, text,
> +                        NGX_SOCKADDR_STRLEN, 0);
> +    if (len == 0) {
> +        return NGX_ERROR;
> +    }
> +
> +    p = ngx_pnalloc(c->pool, len);
> +    if (p == NULL) {
> +        return NGX_ERROR;
> +    }
> +
> +    ngx_memcpy(p, text, len);
> +
> +    c->local_sockaddr = addr_local.sockaddr;
> +    c->local_socklen = addr_local.socklen;

Updating c->local_sockaddr looks like a bad idea even if the 
address comes from trusted proxy servers.  We don't do this 
neither in stream nor in http, and I don't think that mail module 
is a good place to start doing this.

> +
> +    return NGX_OK;
> +}
> +
> +
> +void
> +ngx_mail_proxy_protocol_handler(ngx_event_t *rev)
> +{
> +    ngx_mail_core_srv_conf_t  *cscf;
> +    ngx_mail_session_t        *s;
> +    ngx_connection_t          *c;
> +    u_char                    *p, buf[NGX_PROXY_PROTOCOL_MAX_HEADER];
> +    size_t                     size;
> +    ssize_t                    n;
> +
> +    c = rev->data;
> +    s = c->data;
> +
> +    if (rev->timedout) {
> +        ngx_log_error(NGX_LOG_INFO, c->log, NGX_ETIMEDOUT,
> +                      "mail PROXY protocol header timed out");

There is no need to write such error messages as there is 
c->log->action to provide appropriate context.  Just "client timed 
out" is enough.

> +        c->timedout = 1;
> +        ngx_mail_close_connection(c);
> +        return;
> +    }
> +
> +    ngx_log_debug0(NGX_LOG_DEBUG_MAIL, c->log, 0,
> +                   "mail PROXY protocol handler");
> +
> +    cscf = ngx_mail_get_module_srv_conf(s, ngx_mail_core_module);
> +
> +    if (cscf->realip_from == NGX_CONF_UNSET_PTR) {
> +        ngx_log_error(NGX_LOG_WARN, c->log, 0,
> +                      "using PROXY protocol without set_real_ip_from");
> +        ngx_mail_close_connection(c);

With the approach taken, this is clearly a configuration error. 
Such configurations should be reported (and rejected) during 
configuration parsing.

Further, for configuration errors which cannot be detected during 
configuration parsing yet result in things like connections being 
dropped a better log level is NGX_LOG_ERR.

The approach, however, looks wrong, see above.

> +        return;
> +    }
> +
> +    if (ngx_cidr_match(c->sockaddr, cscf->realip_from) != NGX_OK) {
> +        ngx_log_error(NGX_LOG_NOTICE, c->log, 0,
> +                      "UNTRUSTED PROXY protocol provider: %V",
> +                      &c->addr_text);
> +        ngx_mail_close_connection(c);

The "notice" level also looks wrong here.  Client-related errors 
are usually logged at the "info" level.

The approach, however, looks wrong, see above.

> +        return;
> +    }
> +
> +    size = NGX_PROXY_PROTOCOL_MAX_HEADER;
> +
> +    n = recv(c->fd, (char *) buf, size, MSG_PEEK);
> +
> +    ngx_log_debug1(NGX_LOG_DEBUG, c->log, 0, "mail recv(): %z", n);
> +
> +    p = ngx_proxy_protocol_read(c, buf, buf + n);

This seems to lack any error checking for the "n" value returned 
by recv(), and will happily call ngx_proxy_protocol_read() with 
"last" set to "buf - 1" if recv() returns an error.  Most likely 
this will result in a segmentation fault.

> +
> +    if (p == NULL) {
> +        ngx_mail_close_connection(c);
> +        return;
> +    }
> +
> +    ngx_log_error(NGX_LOG_NOTICE, c->log, 0,
> +                  "PROXY protocol %V:%d => %V:%d",
> +                  &c->proxy_protocol->src_addr,
> +                  c->proxy_protocol->src_port,
> +                  &c->proxy_protocol->dst_addr,
> +                  c->proxy_protocol->dst_port);

Logging level used looks wrong.  At most this should be "info", 
much like connect / disconnect logging.  Or shouldn't be at all, 
as this information is already logged at debug level by 
ngx_proxy_protocol_read().

> +
> +    size = p - buf;
> +
> +    if (c->recv(c, buf, size) != (ssize_t) size) {
> +        ngx_mail_close_connection(c);
> +        return;
> +    }
> +
> +    if (ngx_mail_proxy_protoco_set_addrs(c) != NGX_OK) {
> +        ngx_mail_close_connection(c);
> +        return;
> +    }
> +
> +    ngx_mail_init_connection_complete(c);
> +}
> +
> +
> +void
> +ngx_mail_init_connection_complete(ngx_connection_t *c)
> +{
>  #if (NGX_MAIL_SSL)
>      {
> -    ngx_mail_ssl_conf_t  *sslcf;
> +    ngx_mail_session_t        *s;
> +    ngx_mail_ssl_conf_t       *sslcf;
> +
> +    s = c->data;

Style: since the code under #if is in a separate function now, 
additional block is not needed anymore to introduce variables; 
wrong number of spaces between type and variable names.

>  
>      sslcf = ngx_mail_get_module_srv_conf(s, ngx_mail_ssl_module);
>  
> -    if (sslcf->enable || addr_conf->ssl) {
> +    if (sslcf->enable || s->addr_conf->ssl) {
>          c->log->action = "SSL handshaking";
>  
>          ngx_mail_ssl_init_connection(&sslcf->ssl, c);
> @@ -348,6 +519,7 @@
>          return;
>      }
>  
> +    c->log->action = "sending client greeting line";
>      c->write->handler = ngx_mail_send;
>  
>      cscf->protocol->init_session(s, c);

This looks like a pre-existing bug, probably should be addressed 
by a separate patch.

> diff -r 83c4622053b0 -r 4618e767b84c src/mail/ngx_mail_proxy_module.c
> --- a/src/mail/ngx_mail_proxy_module.c	Tue Jan 12 16:59:31 2021 +0300
> +++ b/src/mail/ngx_mail_proxy_module.c	Tue Jan 19 18:24:23 2021 +0300
> @@ -19,6 +19,8 @@
>      ngx_flag_t  smtp_auth;
>      size_t      buffer_size;
>      ngx_msec_t  timeout;
> +    ngx_msec_t  connect_timeout;
> +    ngx_flag_t  proxy_protocol;
>  } ngx_mail_proxy_conf_t;
>  
>  
> @@ -36,7 +38,9 @@
>  static void *ngx_mail_proxy_create_conf(ngx_conf_t *cf);
>  static char *ngx_mail_proxy_merge_conf(ngx_conf_t *cf, void *parent,
>      void *child);
> -

Style, see above.

> +static void ngx_mail_proxy_connect_handler(ngx_event_t *ev);
> +static void ngx_mail_proxy_start(ngx_mail_session_t *s);
> +static void ngx_mail_proxy_send_proxy_protocol(ngx_mail_session_t *s);

Style: order of function prototypes does not match order of 
functions.

>  
>  static ngx_command_t  ngx_mail_proxy_commands[] = {
>  
> @@ -61,6 +65,13 @@
>        offsetof(ngx_mail_proxy_conf_t, timeout),
>        NULL },
>  
> +    { ngx_string("connect_timeout"),
> +      NGX_MAIL_MAIN_CONF|NGX_MAIL_SRV_CONF|NGX_CONF_TAKE1,
> +      ngx_conf_set_msec_slot,
> +      NGX_MAIL_SRV_CONF_OFFSET,
> +      offsetof(ngx_mail_proxy_conf_t, connect_timeout),
> +      NULL },
> +

Not needed, see above.

>      { ngx_string("proxy_pass_error_message"),
>        NGX_MAIL_MAIN_CONF|NGX_MAIL_SRV_CONF|NGX_CONF_FLAG,
>        ngx_conf_set_flag_slot,
> @@ -82,6 +93,13 @@
>        offsetof(ngx_mail_proxy_conf_t, smtp_auth),
>        NULL },
>  
> +    { ngx_string("proxy_protocol"),
> +      NGX_MAIL_MAIN_CONF|NGX_MAIL_SRV_CONF|NGX_CONF_FLAG,
> +      ngx_conf_set_flag_slot,
> +      NGX_MAIL_SRV_CONF_OFFSET,
> +      offsetof(ngx_mail_proxy_conf_t, proxy_protocol),
> +      NULL },
> +
>        ngx_null_command
>  };
>  
> @@ -156,7 +174,6 @@
>      p->upstream.connection->pool = s->connection->pool;
>  
>      s->connection->read->handler = ngx_mail_proxy_block_read;
> -    p->upstream.connection->write->handler = ngx_mail_proxy_dummy_handler;
>  
>      pcf = ngx_mail_get_module_srv_conf(s, ngx_mail_proxy_module);
>  
> @@ -169,23 +186,139 @@
>  
>      s->out.len = 0;
>  
> +    if (rc == NGX_AGAIN) {
> +        p->upstream.connection->write->handler = ngx_mail_proxy_connect_handler;
> +        p->upstream.connection->read->handler = ngx_mail_proxy_connect_handler;
> +
> +        ngx_add_timer(p->upstream.connection->write, pcf->connect_timeout);
> +
> +        ngx_log_debug0(NGX_LOG_DEBUG_MAIL, s->connection->log, 0, "mail proxy delay connect");

Style: too long line, should be less than 80 chars.

Further, the message looks misleading.  There is no delay here, 
but rather connect is in progress.

If you think that better debugging should be added to indicate the 
connect() status, a better approach would be to add debugging 
right after the ngx_event_connect_peer() call, like in the 
ngx_http_upstream_module:

    rc = ngx_event_connect_peer(&u->peer);

    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "http upstream connect: %i", rc);

Note though that this look unrelated to the change itself, and 
should be a separate patch.

> +        return;
> +    }
> +
> +    if (pcf->proxy_protocol) {
> +        ngx_mail_proxy_send_proxy_protocol(s);
> +        return;
> +    }
> +
> +    ngx_mail_proxy_start(s);
> +}
> +
> +
> +void
> +ngx_mail_proxy_connect_handler(ngx_event_t *ev)
> +{
> +    ngx_connection_t          *c;
> +    ngx_mail_session_t        *s;
> +    ngx_mail_proxy_conf_t     *pcf;

Style: should be two spaces between the longest type and 
variable name.

> +
> +    c = ev->data;
> +    s = c->data;
> +
> +    if (ev->timedout) {
> +        ngx_log_error(NGX_LOG_ERR, c->log, NGX_ETIMEDOUT, "upstream timed out");
> +        ngx_mail_session_internal_server_error(s);
> +        return;
> +    }
> +
> +    ngx_del_timer(c->write);
> +
> +    ngx_log_debug0(NGX_LOG_DEBUG_MAIL, c->log, 0,
> +                   "mail proxy connect upstream");
> +
> +    pcf = ngx_mail_get_module_srv_conf(s, ngx_mail_proxy_module);
> +
> +    if (pcf->proxy_protocol) {
> +        ngx_mail_proxy_send_proxy_protocol(s);
> +        return;
> +    }
> +
> +    ngx_mail_proxy_start(s);
> +}
> +
> +
> +void
> +ngx_mail_proxy_start(ngx_mail_session_t *s)
> +{
> +    ngx_connection_t             *pc;
> +
> +    pc = s->proxy->upstream.connection;
> +
> +    ngx_log_debug0(NGX_LOG_DEBUG_MAIL, s->connection->log, 0,
> +                   "mail proxy starting");
> +
> +    pc->write->handler = ngx_mail_proxy_dummy_handler;
> +
>      switch (s->protocol) {
>  
>      case NGX_MAIL_POP3_PROTOCOL:
> -        p->upstream.connection->read->handler = ngx_mail_proxy_pop3_handler;
> +        pc->read->handler = ngx_mail_proxy_pop3_handler;
>          s->mail_state = ngx_pop3_start;
>          break;
>  
>      case NGX_MAIL_IMAP_PROTOCOL:
> -        p->upstream.connection->read->handler = ngx_mail_proxy_imap_handler;
> +        pc->read->handler = ngx_mail_proxy_imap_handler;
>          s->mail_state = ngx_imap_start;
>          break;
>  
>      default: /* NGX_MAIL_SMTP_PROTOCOL */
> -        p->upstream.connection->read->handler = ngx_mail_proxy_smtp_handler;
> +        pc->read->handler = ngx_mail_proxy_smtp_handler;
>          s->mail_state = ngx_smtp_start;
>          break;
>      }
> +
> +    if (pc->read->ready) {
> +        ngx_post_event(pc->read, &ngx_posted_events);
> +    }
> +}
> +
> +
> +void
> +ngx_mail_proxy_send_proxy_protocol(ngx_mail_session_t *s)
> +{
> +    u_char                       *p;
> +    ssize_t                       n, size;
> +    ngx_connection_t             *c, *pc;
> +    ngx_peer_connection_t        *u;
> +    u_char                        buf[NGX_PROXY_PROTOCOL_MAX_HEADER];
> +
> +    c = s->connection;
> +
> +    ngx_log_debug0(NGX_LOG_DEBUG_MAIL, c->log, 0,
> +                   "mail proxy send PROXY protocol header");
> +
> +    p = ngx_proxy_protocol_write(c, buf, buf + NGX_PROXY_PROTOCOL_MAX_HEADER);
> +    if (p == NULL) {
> +        ngx_mail_proxy_internal_server_error(s);
> +        return;
> +    }
> +
> +    u = &s->proxy->upstream;
> +
> +    pc = u->connection;
> +
> +    size = p - buf;
> +
> +    n = pc->send(pc, buf, size);
> +
> +    if (n != size) {
> +
> +        /*
> +         * PROXY protocol specification:
> +         * The sender must always ensure that the header
> +         * is sent at once, so that the transport layer
> +         * maintains atomicity along the path to the receiver.
> +         */
> +
> +        ngx_log_error(NGX_LOG_ERR, c->log, 0,
> +                      "could not send PROXY protocol header at once (%z)", n);
> +
> +        ngx_mail_proxy_internal_server_error(s);
> +
> +        return;
> +    }

The error handling seems to be incomplete here.  While it is 
unlikely that things like NGX_AGAIN will be returned here, it is 
quite possible that c->send() will fail with an error if 
connection is reset by the upstream server, and this is going to 
result in duplicate and misleading error message.

> +
> +    ngx_mail_proxy_start(s);
>  }
>  
>  
> @@ -1184,6 +1317,8 @@
>      pcf->smtp_auth = NGX_CONF_UNSET;
>      pcf->buffer_size = NGX_CONF_UNSET_SIZE;
>      pcf->timeout = NGX_CONF_UNSET_MSEC;
> +    pcf->connect_timeout = NGX_CONF_UNSET_MSEC;
> +    pcf->proxy_protocol = NGX_CONF_UNSET;
>  
>      return pcf;
>  }
> @@ -1202,6 +1337,8 @@
>      ngx_conf_merge_size_value(conf->buffer_size, prev->buffer_size,
>                                (size_t) ngx_pagesize);
>      ngx_conf_merge_msec_value(conf->timeout, prev->timeout, 24 * 60 * 60000);
> +    ngx_conf_merge_msec_value(conf->connect_timeout, prev->connect_timeout, 1000);
> +    ngx_conf_merge_value(conf->proxy_protocol, prev->proxy_protocol, 0);
>  
>      return NGX_CONF_OK;
>  }

Below is a patch series which tries to address most of the above 
comments, and also fixes several existing problems in the code 
found in the process.  Review is appreciated.

# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1613784928 -10800
#      Sat Feb 20 04:35:28 2021 +0300
# Node ID 00bc617ccb86019850640db9c93ee6d8ea5f56d4
# Parent  f77ad78046dcaa39ae4d4ddeeb52a63846f7d579
SSL: fixed build by Sun C with old OpenSSL versions.

Sun C complains about "statement not reached" if a "return" is followed
by additional statements.

diff --git a/src/http/modules/ngx_http_grpc_module.c b/src/http/modules/ngx_http_grpc_module.c
--- a/src/http/modules/ngx_http_grpc_module.c
+++ b/src/http/modules/ngx_http_grpc_module.c
@@ -4841,9 +4841,9 @@ ngx_http_grpc_ssl_conf_command_check(ngx
 {
 #ifndef SSL_CONF_FLAG_FILE
     return "is not supported on this platform";
+#else
+    return NGX_CONF_OK;
 #endif
-
-    return NGX_CONF_OK;
 }
 
 
diff --git a/src/http/modules/ngx_http_proxy_module.c b/src/http/modules/ngx_http_proxy_module.c
--- a/src/http/modules/ngx_http_proxy_module.c
+++ b/src/http/modules/ngx_http_proxy_module.c
@@ -4876,9 +4876,9 @@ ngx_http_proxy_ssl_conf_command_check(ng
 {
 #ifndef SSL_CONF_FLAG_FILE
     return "is not supported on this platform";
+#else
+    return NGX_CONF_OK;
 #endif
-
-    return NGX_CONF_OK;
 }
 
 
diff --git a/src/http/modules/ngx_http_ssl_module.c b/src/http/modules/ngx_http_ssl_module.c
--- a/src/http/modules/ngx_http_ssl_module.c
+++ b/src/http/modules/ngx_http_ssl_module.c
@@ -1274,9 +1274,9 @@ ngx_http_ssl_conf_command_check(ngx_conf
 {
 #ifndef SSL_CONF_FLAG_FILE
     return "is not supported on this platform";
+#else
+    return NGX_CONF_OK;
 #endif
-
-    return NGX_CONF_OK;
 }
 
 
diff --git a/src/http/modules/ngx_http_uwsgi_module.c b/src/http/modules/ngx_http_uwsgi_module.c
--- a/src/http/modules/ngx_http_uwsgi_module.c
+++ b/src/http/modules/ngx_http_uwsgi_module.c
@@ -2398,9 +2398,9 @@ ngx_http_uwsgi_ssl_conf_command_check(ng
 {
 #ifndef SSL_CONF_FLAG_FILE
     return "is not supported on this platform";
+#else
+    return NGX_CONF_OK;
 #endif
-
-    return NGX_CONF_OK;
 }
 
 
diff --git a/src/mail/ngx_mail_ssl_module.c b/src/mail/ngx_mail_ssl_module.c
--- a/src/mail/ngx_mail_ssl_module.c
+++ b/src/mail/ngx_mail_ssl_module.c
@@ -682,7 +682,7 @@ ngx_mail_ssl_conf_command_check(ngx_conf
 {
 #ifndef SSL_CONF_FLAG_FILE
     return "is not supported on this platform";
+#else
+    return NGX_CONF_OK;
 #endif
-
-    return NGX_CONF_OK;
 }
diff --git a/src/stream/ngx_stream_proxy_module.c b/src/stream/ngx_stream_proxy_module.c
--- a/src/stream/ngx_stream_proxy_module.c
+++ b/src/stream/ngx_stream_proxy_module.c
@@ -1026,9 +1026,9 @@ ngx_stream_proxy_ssl_conf_command_check(
 {
 #ifndef SSL_CONF_FLAG_FILE
     return "is not supported on this platform";
+#else
+    return NGX_CONF_OK;
 #endif
-
-    return NGX_CONF_OK;
 }
 
 
diff --git a/src/stream/ngx_stream_ssl_module.c b/src/stream/ngx_stream_ssl_module.c
--- a/src/stream/ngx_stream_ssl_module.c
+++ b/src/stream/ngx_stream_ssl_module.c
@@ -1061,9 +1061,9 @@ ngx_stream_ssl_conf_command_check(ngx_co
 {
 #ifndef SSL_CONF_FLAG_FILE
     return "is not supported on this platform";
+#else
+    return NGX_CONF_OK;
 #endif
-
-    return NGX_CONF_OK;
 }
 
 
# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1613784931 -10800
#      Sat Feb 20 04:35:31 2021 +0300
# Node ID 63777b5cbc69027eb96ff70674d4ffa2b99aa3a4
# Parent  00bc617ccb86019850640db9c93ee6d8ea5f56d4
Events: fixed eventport handling in ngx_handle_read_event().

The "!rev->ready" test seems to be a typo, introduced in the original
commit (719:f30b1a75fd3b).  The ngx_handle_write_event() code properly
tests for "rev->ready" instead.

Due to this typo, read events might be unexpectedly removed during
proxying after an event on the other part of the proxied connection.
Catched by mail proxying tests.

diff --git a/src/event/ngx_event.c b/src/event/ngx_event.c
--- a/src/event/ngx_event.c
+++ b/src/event/ngx_event.c
@@ -318,7 +318,7 @@ ngx_handle_read_event(ngx_event_t *rev, 
             return NGX_OK;
         }
 
-        if (rev->oneshot && !rev->ready) {
+        if (rev->oneshot && rev->ready) {
             if (ngx_del_event(rev, NGX_READ_EVENT, 0) == NGX_ERROR) {
                 return NGX_ERROR;
             }
# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1613784935 -10800
#      Sat Feb 20 04:35:35 2021 +0300
# Node ID f776e728c2527301bf2cd22ef11bee1fb982f537
# Parent  63777b5cbc69027eb96ff70674d4ffa2b99aa3a4
Mail: added missing event handling after blocking events.

As long as a read event is blocked (ignored), ngx_handle_read_event()
needs to be called to make sure no further notifications will be
triggered when using level-triggered event methods, such as select() or
poll().

diff --git a/src/mail/ngx_mail_imap_handler.c b/src/mail/ngx_mail_imap_handler.c
--- a/src/mail/ngx_mail_imap_handler.c
+++ b/src/mail/ngx_mail_imap_handler.c
@@ -123,6 +123,12 @@ ngx_mail_imap_auth_state(ngx_event_t *re
     if (s->out.len) {
         ngx_log_debug0(NGX_LOG_DEBUG_MAIL, c->log, 0, "imap send handler busy");
         s->blocked = 1;
+
+        if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
+            ngx_mail_close_connection(c);
+            return;
+        }
+
         return;
     }
 
diff --git a/src/mail/ngx_mail_pop3_handler.c b/src/mail/ngx_mail_pop3_handler.c
--- a/src/mail/ngx_mail_pop3_handler.c
+++ b/src/mail/ngx_mail_pop3_handler.c
@@ -138,6 +138,12 @@ ngx_mail_pop3_auth_state(ngx_event_t *re
     if (s->out.len) {
         ngx_log_debug0(NGX_LOG_DEBUG_MAIL, c->log, 0, "pop3 send handler busy");
         s->blocked = 1;
+
+        if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
+            ngx_mail_close_connection(c);
+            return;
+        }
+
         return;
     }
 
diff --git a/src/mail/ngx_mail_smtp_handler.c b/src/mail/ngx_mail_smtp_handler.c
--- a/src/mail/ngx_mail_smtp_handler.c
+++ b/src/mail/ngx_mail_smtp_handler.c
@@ -449,6 +449,12 @@ ngx_mail_smtp_auth_state(ngx_event_t *re
     if (s->out.len) {
         ngx_log_debug0(NGX_LOG_DEBUG_MAIL, c->log, 0, "smtp send handler busy");
         s->blocked = 1;
+
+        if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
+            ngx_mail_close_connection(c);
+            return;
+        }
+
         return;
     }
 
# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1613787594 -10800
#      Sat Feb 20 05:19:54 2021 +0300
# Node ID 0ab90ee2fd58ec5d33405333cc44049d8e5a37a7
# Parent  f776e728c2527301bf2cd22ef11bee1fb982f537
Mail: added missing event handling after reading data.

If we need to be notified about further events, ngx_handle_read_event()
needs to be called after a read event is processed.  Without this,
an event can be removed from the kernel and won't be reported again,
notably when using oneshot event methods, such as eventport on Solaris.

For consistency, existing ngx_handle_read_event() call removed from
ngx_mail_read_command(), as this call only covers of of the code paths
where ngx_mail_read_command() returns NGX_AGAIN.  Instead, appropriate
processing added to the callers, covering all code paths where NGX_AGAIN
is returned.

diff --git a/src/mail/ngx_mail_handler.c b/src/mail/ngx_mail_handler.c
--- a/src/mail/ngx_mail_handler.c
+++ b/src/mail/ngx_mail_handler.c
@@ -722,11 +722,6 @@ ngx_mail_read_command(ngx_mail_session_t
     }
 
     if (n == NGX_AGAIN) {
-        if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
-            ngx_mail_session_internal_server_error(s);
-            return NGX_ERROR;
-        }
-
         if (s->buffer->pos == s->buffer->last) {
             return NGX_AGAIN;
         }
diff --git a/src/mail/ngx_mail_imap_handler.c b/src/mail/ngx_mail_imap_handler.c
--- a/src/mail/ngx_mail_imap_handler.c
+++ b/src/mail/ngx_mail_imap_handler.c
@@ -136,7 +136,16 @@ ngx_mail_imap_auth_state(ngx_event_t *re
 
     rc = ngx_mail_read_command(s, c);
 
-    if (rc == NGX_AGAIN || rc == NGX_ERROR) {
+    if (rc == NGX_AGAIN) {
+        if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
+            ngx_mail_session_internal_server_error(s);
+            return;
+        }
+
+        return;
+    }
+
+    if (rc == NGX_ERROR) {
         return;
     }
 
@@ -299,6 +308,11 @@ ngx_mail_imap_auth_state(ngx_event_t *re
         }
     }
 
+    if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
+        ngx_mail_session_internal_server_error(s);
+        return;
+    }
+
     ngx_mail_send(c->write);
 }
 
diff --git a/src/mail/ngx_mail_pop3_handler.c b/src/mail/ngx_mail_pop3_handler.c
--- a/src/mail/ngx_mail_pop3_handler.c
+++ b/src/mail/ngx_mail_pop3_handler.c
@@ -151,7 +151,16 @@ ngx_mail_pop3_auth_state(ngx_event_t *re
 
     rc = ngx_mail_read_command(s, c);
 
-    if (rc == NGX_AGAIN || rc == NGX_ERROR) {
+    if (rc == NGX_AGAIN) {
+        if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
+            ngx_mail_session_internal_server_error(s);
+            return;
+        }
+
+        return;
+    }
+
+    if (rc == NGX_ERROR) {
         return;
     }
 
@@ -281,6 +290,11 @@ ngx_mail_pop3_auth_state(ngx_event_t *re
             s->arg_start = s->buffer->start;
         }
 
+        if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
+            ngx_mail_session_internal_server_error(s);
+            return;
+        }
+
         ngx_mail_send(c->write);
     }
 }
diff --git a/src/mail/ngx_mail_proxy_module.c b/src/mail/ngx_mail_proxy_module.c
--- a/src/mail/ngx_mail_proxy_module.c
+++ b/src/mail/ngx_mail_proxy_module.c
@@ -233,6 +233,11 @@ ngx_mail_proxy_pop3_handler(ngx_event_t 
     rc = ngx_mail_proxy_read_response(s, 0);
 
     if (rc == NGX_AGAIN) {
+        if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
+            ngx_mail_proxy_internal_server_error(s);
+            return;
+        }
+
         return;
     }
 
@@ -314,6 +319,11 @@ ngx_mail_proxy_pop3_handler(ngx_event_t 
         return;
     }
 
+    if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
+        ngx_mail_proxy_internal_server_error(s);
+        return;
+    }
+
     s->proxy->buffer->pos = s->proxy->buffer->start;
     s->proxy->buffer->last = s->proxy->buffer->start;
 }
@@ -346,6 +356,11 @@ ngx_mail_proxy_imap_handler(ngx_event_t 
     rc = ngx_mail_proxy_read_response(s, s->mail_state);
 
     if (rc == NGX_AGAIN) {
+        if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
+            ngx_mail_proxy_internal_server_error(s);
+            return;
+        }
+
         return;
     }
 
@@ -448,6 +463,11 @@ ngx_mail_proxy_imap_handler(ngx_event_t 
         return;
     }
 
+    if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
+        ngx_mail_proxy_internal_server_error(s);
+        return;
+    }
+
     s->proxy->buffer->pos = s->proxy->buffer->start;
     s->proxy->buffer->last = s->proxy->buffer->start;
 }
@@ -482,6 +502,11 @@ ngx_mail_proxy_smtp_handler(ngx_event_t 
     rc = ngx_mail_proxy_read_response(s, s->mail_state);
 
     if (rc == NGX_AGAIN) {
+        if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
+            ngx_mail_proxy_internal_server_error(s);
+            return;
+        }
+
         return;
     }
 
@@ -763,6 +788,11 @@ ngx_mail_proxy_smtp_handler(ngx_event_t 
         return;
     }
 
+    if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
+        ngx_mail_proxy_internal_server_error(s);
+        return;
+    }
+
     s->proxy->buffer->pos = s->proxy->buffer->start;
     s->proxy->buffer->last = s->proxy->buffer->start;
 }
diff --git a/src/mail/ngx_mail_smtp_handler.c b/src/mail/ngx_mail_smtp_handler.c
--- a/src/mail/ngx_mail_smtp_handler.c
+++ b/src/mail/ngx_mail_smtp_handler.c
@@ -462,7 +462,16 @@ ngx_mail_smtp_auth_state(ngx_event_t *re
 
     rc = ngx_mail_read_command(s, c);
 
-    if (rc == NGX_AGAIN || rc == NGX_ERROR) {
+    if (rc == NGX_AGAIN) {
+        if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
+            ngx_mail_session_internal_server_error(s);
+            return;
+        }
+
+        return;
+    }
+
+    if (rc == NGX_ERROR) {
         return;
     }
 
@@ -574,6 +583,11 @@ ngx_mail_smtp_auth_state(ngx_event_t *re
             s->arg_start = s->buffer->pos;
         }
 
+        if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
+            ngx_mail_session_internal_server_error(s);
+            return;
+        }
+
         ngx_mail_send(c->write);
     }
 }
# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1613787635 -10800
#      Sat Feb 20 05:20:35 2021 +0300
# Node ID 7361e60def463335d33d6406257b2f2f905d33dd
# Parent  0ab90ee2fd58ec5d33405333cc44049d8e5a37a7
Mail: postponed session initialization under accept mutex.

Similarly to 40e8ce405859 in the stream module, this reduces the time
accept mutex is held.  This also simplifies following changes to
introduce PROXY protocol support.

diff --git a/src/mail/ngx_mail.h b/src/mail/ngx_mail.h
--- a/src/mail/ngx_mail.h
+++ b/src/mail/ngx_mail.h
@@ -197,6 +197,7 @@ typedef struct {
 
     ngx_uint_t              mail_state;
 
+    unsigned                ssl:1;
     unsigned                protocol:3;
     unsigned                blocked:1;
     unsigned                quit:1;
diff --git a/src/mail/ngx_mail_handler.c b/src/mail/ngx_mail_handler.c
--- a/src/mail/ngx_mail_handler.c
+++ b/src/mail/ngx_mail_handler.c
@@ -11,6 +11,7 @@
 #include <ngx_mail.h>
 
 
+static void ngx_mail_init_session_handler(ngx_event_t *rev);
 static void ngx_mail_init_session(ngx_connection_t *c);
 
 #if (NGX_MAIL_SSL)
@@ -26,6 +27,7 @@ ngx_mail_init_connection(ngx_connection_
 {
     size_t                     len;
     ngx_uint_t                 i;
+    ngx_event_t               *rev;
     ngx_mail_port_t           *port;
     struct sockaddr           *sa;
     struct sockaddr_in        *sin;
@@ -129,6 +131,10 @@ ngx_mail_init_connection(ngx_connection_
     s->main_conf = addr_conf->ctx->main_conf;
     s->srv_conf = addr_conf->ctx->srv_conf;
 
+#if (NGX_MAIL_SSL)
+    s->ssl = addr_conf->ssl;
+#endif
+
     s->addr_text = &addr_conf->addr_text;
 
     c->data = s;
@@ -159,13 +165,34 @@ ngx_mail_init_connection(ngx_connection_
 
     c->log_error = NGX_ERROR_INFO;
 
+    rev = c->read;
+    rev->handler = ngx_mail_init_session_handler;
+
+    if (ngx_use_accept_mutex) {
+        ngx_post_event(rev, &ngx_posted_events);
+        return;
+    }
+
+    rev->handler(rev);
+}
+
+
+static void
+ngx_mail_init_session_handler(ngx_event_t *rev)
+{
+    ngx_connection_t    *c;
+    ngx_mail_session_t  *s;
+
+    c = rev->data;
+    s = c->data;
+
 #if (NGX_MAIL_SSL)
     {
     ngx_mail_ssl_conf_t  *sslcf;
 
     sslcf = ngx_mail_get_module_srv_conf(s, ngx_mail_ssl_module);
 
-    if (sslcf->enable || addr_conf->ssl) {
+    if (sslcf->enable || s->ssl) {
         c->log->action = "SSL handshaking";
 
         ngx_mail_ssl_init_connection(&sslcf->ssl, c);
# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1613787636 -10800
#      Sat Feb 20 05:20:36 2021 +0300
# Node ID 698118019f618a4b746986b727189e281310d9d1
# Parent  7361e60def463335d33d6406257b2f2f905d33dd
Mail: fixed log action after SSL handshake.

diff --git a/src/mail/ngx_mail_handler.c b/src/mail/ngx_mail_handler.c
--- a/src/mail/ngx_mail_handler.c
+++ b/src/mail/ngx_mail_handler.c
@@ -365,6 +365,8 @@ ngx_mail_init_session(ngx_connection_t *
 
     s = c->data;
 
+    c->log->action = "sending client greeting line";
+
     cscf = ngx_mail_get_module_srv_conf(s, ngx_mail_core_module);
 
     s->protocol = cscf->protocol->type;
# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1613787637 -10800
#      Sat Feb 20 05:20:37 2021 +0300
# Node ID 23008e1a0113f5bc22028473e1c914e32d503374
# Parent  698118019f618a4b746986b727189e281310d9d1
Mail: made auth http creating request easier to extend.

diff --git a/src/mail/ngx_mail_auth_http_module.c b/src/mail/ngx_mail_auth_http_module.c
--- a/src/mail/ngx_mail_auth_http_module.c
+++ b/src/mail/ngx_mail_auth_http_module.c
@@ -1224,22 +1224,38 @@ ngx_mail_auth_http_create_request(ngx_ma
           + sizeof("Client-IP: ") - 1 + s->connection->addr_text.len
                 + sizeof(CRLF) - 1
           + sizeof("Client-Host: ") - 1 + s->host.len + sizeof(CRLF) - 1
-          + sizeof("Auth-SMTP-Helo: ") - 1 + s->smtp_helo.len + sizeof(CRLF) - 1
-          + sizeof("Auth-SMTP-From: ") - 1 + s->smtp_from.len + sizeof(CRLF) - 1
-          + sizeof("Auth-SMTP-To: ") - 1 + s->smtp_to.len + sizeof(CRLF) - 1
-#if (NGX_MAIL_SSL)
-          + sizeof("Auth-SSL: on" CRLF) - 1
-          + sizeof("Auth-SSL-Verify: ") - 1 + verify.len + sizeof(CRLF) - 1
-          + sizeof("Auth-SSL-Subject: ") - 1 + subject.len + sizeof(CRLF) - 1
-          + sizeof("Auth-SSL-Issuer: ") - 1 + issuer.len + sizeof(CRLF) - 1
-          + sizeof("Auth-SSL-Serial: ") - 1 + serial.len + sizeof(CRLF) - 1
-          + sizeof("Auth-SSL-Fingerprint: ") - 1 + fingerprint.len
-              + sizeof(CRLF) - 1
-          + sizeof("Auth-SSL-Cert: ") - 1 + cert.len + sizeof(CRLF) - 1
-#endif
           + ahcf->header.len
           + sizeof(CRLF) - 1;
 
+    if (s->auth_method == NGX_MAIL_AUTH_NONE) {
+        len += sizeof("Auth-SMTP-Helo: ") - 1 + s->smtp_helo.len
+                     + sizeof(CRLF) - 1
+               + sizeof("Auth-SMTP-From: ") - 1 + s->smtp_from.len
+                     + sizeof(CRLF) - 1
+               + sizeof("Auth-SMTP-To: ") - 1 + s->smtp_to.len
+                     + sizeof(CRLF) - 1;
+    }
+
+#if (NGX_MAIL_SSL)
+
+    if (c->ssl) {
+        len += sizeof("Auth-SSL: on" CRLF) - 1
+               + sizeof("Auth-SSL-Verify: ") - 1 + verify.len
+                     + sizeof(CRLF) - 1
+               + sizeof("Auth-SSL-Subject: ") - 1 + subject.len
+                     + sizeof(CRLF) - 1
+               + sizeof("Auth-SSL-Issuer: ") - 1 + issuer.len
+                     + sizeof(CRLF) - 1
+               + sizeof("Auth-SSL-Serial: ") - 1 + serial.len
+                     + sizeof(CRLF) - 1
+               + sizeof("Auth-SSL-Fingerprint: ") - 1 + fingerprint.len
+                     + sizeof(CRLF) - 1
+               + sizeof("Auth-SSL-Cert: ") - 1 + cert.len
+                     + sizeof(CRLF) - 1;
+    }
+
+#endif
+
     b = ngx_create_temp_buf(pool, len);
     if (b == NULL) {
         return NULL;
# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1613793031 -10800
#      Sat Feb 20 06:50:31 2021 +0300
# Node ID e7035246851a7c50ac7083f563ba754cc3ddb2f0
# Parent  23008e1a0113f5bc22028473e1c914e32d503374
Mail: parsing of the PROXY protocol from clients.

Activated with the "proxy_protocol" parameter of the "listen" directive.
Obtained information is passed to the auth_http script in Proxy-Protocol-Addr,
Proxy-Protocol-Port, Proxy-Protocol-Server-Addr, and Proxy-Protocol-Server-Port
headers.

diff --git a/src/mail/ngx_mail.c b/src/mail/ngx_mail.c
--- a/src/mail/ngx_mail.c
+++ b/src/mail/ngx_mail.c
@@ -405,6 +405,7 @@ ngx_mail_add_addrs(ngx_conf_t *cf, ngx_m
 #if (NGX_MAIL_SSL)
         addrs[i].conf.ssl = addr[i].opt.ssl;
 #endif
+        addrs[i].conf.proxy_protocol = addr[i].opt.proxy_protocol;
         addrs[i].conf.addr_text = addr[i].opt.addr_text;
     }
 
@@ -439,6 +440,7 @@ ngx_mail_add_addrs6(ngx_conf_t *cf, ngx_
 #if (NGX_MAIL_SSL)
         addrs6[i].conf.ssl = addr[i].opt.ssl;
 #endif
+        addrs6[i].conf.proxy_protocol = addr[i].opt.proxy_protocol;
         addrs6[i].conf.addr_text = addr[i].opt.addr_text;
     }
 
diff --git a/src/mail/ngx_mail.h b/src/mail/ngx_mail.h
--- a/src/mail/ngx_mail.h
+++ b/src/mail/ngx_mail.h
@@ -41,6 +41,7 @@ typedef struct {
     unsigned                ipv6only:1;
 #endif
     unsigned                so_keepalive:2;
+    unsigned                proxy_protocol:1;
 #if (NGX_HAVE_KEEPALIVE_TUNABLE)
     int                     tcp_keepidle;
     int                     tcp_keepintvl;
@@ -55,7 +56,8 @@ typedef struct {
 typedef struct {
     ngx_mail_conf_ctx_t    *ctx;
     ngx_str_t               addr_text;
-    ngx_uint_t              ssl;    /* unsigned   ssl:1; */
+    unsigned                ssl:1;
+    unsigned                proxy_protocol:1;
 } ngx_mail_addr_conf_t;
 
 typedef struct {
diff --git a/src/mail/ngx_mail_auth_http_module.c b/src/mail/ngx_mail_auth_http_module.c
--- a/src/mail/ngx_mail_auth_http_module.c
+++ b/src/mail/ngx_mail_auth_http_module.c
@@ -1227,6 +1227,17 @@ ngx_mail_auth_http_create_request(ngx_ma
           + ahcf->header.len
           + sizeof(CRLF) - 1;
 
+    if (c->proxy_protocol) {
+        len += sizeof("Proxy-Protocol-Addr: ") - 1
+                     + c->proxy_protocol->src_addr.len + sizeof(CRLF) - 1
+               + sizeof("Proxy-Protocol-Port: ") - 1
+                     + sizeof("65535") - 1 + sizeof(CRLF) - 1
+               + sizeof("Proxy-Protocol-Server-Addr: ") - 1
+                     + c->proxy_protocol->dst_addr.len + sizeof(CRLF) - 1
+               + sizeof("Proxy-Protocol-Server-Port: ") - 1
+                     + sizeof("65535") - 1 + sizeof(CRLF) - 1;
+    }
+
     if (s->auth_method == NGX_MAIL_AUTH_NONE) {
         len += sizeof("Auth-SMTP-Helo: ") - 1 + s->smtp_helo.len
                      + sizeof(CRLF) - 1
@@ -1314,6 +1325,26 @@ ngx_mail_auth_http_create_request(ngx_ma
         *b->last++ = CR; *b->last++ = LF;
     }
 
+    if (c->proxy_protocol) {
+        b->last = ngx_cpymem(b->last, "Proxy-Protocol-Addr: ",
+                             sizeof("Proxy-Protocol-Addr: ") - 1);
+        b->last = ngx_copy(b->last, c->proxy_protocol->src_addr.data,
+                           c->proxy_protocol->src_addr.len);
+        *b->last++ = CR; *b->last++ = LF;
+
+        b->last = ngx_sprintf(b->last, "Proxy-Protocol-Port: %d" CRLF,
+                              c->proxy_protocol->src_port);
+
+        b->last = ngx_cpymem(b->last, "Proxy-Protocol-Server-Addr: ",
+                             sizeof("Proxy-Protocol-Server-Addr: ") - 1);
+        b->last = ngx_copy(b->last, c->proxy_protocol->dst_addr.data,
+                           c->proxy_protocol->dst_addr.len);
+        *b->last++ = CR; *b->last++ = LF;
+
+        b->last = ngx_sprintf(b->last, "Proxy-Protocol-Server-Port: %d" CRLF,
+                              c->proxy_protocol->dst_port);
+    }
+
     if (s->auth_method == NGX_MAIL_AUTH_NONE) {
 
         /* HELO, MAIL FROM, and RCPT TO can't contain CRLF, no need to escape */
diff --git a/src/mail/ngx_mail_core_module.c b/src/mail/ngx_mail_core_module.c
--- a/src/mail/ngx_mail_core_module.c
+++ b/src/mail/ngx_mail_core_module.c
@@ -548,6 +548,11 @@ ngx_mail_core_listen(ngx_conf_t *cf, ngx
 #endif
         }
 
+        if (ngx_strcmp(value[i].data, "proxy_protocol") == 0) {
+            ls->proxy_protocol = 1;
+            continue;
+        }
+
         ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                            "the invalid \"%V\" parameter", &value[i]);
         return NGX_CONF_ERROR;
diff --git a/src/mail/ngx_mail_handler.c b/src/mail/ngx_mail_handler.c
--- a/src/mail/ngx_mail_handler.c
+++ b/src/mail/ngx_mail_handler.c
@@ -11,6 +11,7 @@
 #include <ngx_mail.h>
 
 
+static void ngx_mail_proxy_protocol_handler(ngx_event_t *rev);
 static void ngx_mail_init_session_handler(ngx_event_t *rev);
 static void ngx_mail_init_session(ngx_connection_t *c);
 
@@ -168,6 +169,22 @@ ngx_mail_init_connection(ngx_connection_
     rev = c->read;
     rev->handler = ngx_mail_init_session_handler;
 
+    if (addr_conf->proxy_protocol) {
+        c->log->action = "reading PROXY protocol";
+
+        rev->handler = ngx_mail_proxy_protocol_handler;
+
+        if (!rev->ready) {
+            ngx_add_timer(rev, cscf->timeout);
+
+            if (ngx_handle_read_event(rev, 0) != NGX_OK) {
+                ngx_mail_close_connection(c);
+            }
+
+            return;
+        }
+    }
+
     if (ngx_use_accept_mutex) {
         ngx_post_event(rev, &ngx_posted_events);
         return;
@@ -178,6 +195,76 @@ ngx_mail_init_connection(ngx_connection_
 
 
 static void
+ngx_mail_proxy_protocol_handler(ngx_event_t *rev)
+{
+    u_char                    *p, buf[NGX_PROXY_PROTOCOL_MAX_HEADER];
+    size_t                     size;
+    ssize_t                    n;
+    ngx_err_t                  err;
+    ngx_connection_t          *c;
+    ngx_mail_session_t        *s;
+    ngx_mail_core_srv_conf_t  *cscf;
+
+    c = rev->data;
+    s = c->data;
+
+    ngx_log_debug0(NGX_LOG_DEBUG_MAIL, c->log, 0,
+                   "mail PROXY protocol handler");
+
+    if (rev->timedout) {
+        ngx_log_error(NGX_LOG_INFO, c->log, NGX_ETIMEDOUT, "client timed out");
+        c->timedout = 1;
+        ngx_mail_close_connection(c);
+        return;
+    }
+
+    n = recv(c->fd, (char *) buf, sizeof(buf), MSG_PEEK);
+
+    err = ngx_socket_errno;
+
+    ngx_log_debug1(NGX_LOG_DEBUG_MAIL, c->log, 0, "recv(): %z", n);
+
+    if (n == -1) {
+        if (err == NGX_EAGAIN) {
+            rev->ready = 0;
+
+            if (!rev->timer_set) {
+                cscf = ngx_mail_get_module_srv_conf(s, ngx_mail_core_module);
+                ngx_add_timer(rev, cscf->timeout);
+            }
+
+            if (ngx_handle_read_event(rev, 0) != NGX_OK) {
+                ngx_mail_close_connection(c);
+            }
+
+            return;
+        }
+
+        ngx_connection_error(c, err, "recv() failed");
+
+        ngx_mail_close_connection(c);
+        return;
+    }
+
+    p = ngx_proxy_protocol_read(c, buf, buf + n);
+
+    if (p == NULL) {
+        ngx_mail_close_connection(c);
+        return;
+    }
+
+    size = p - buf;
+
+    if (c->recv(c, buf, size) != (ssize_t) size) {
+        ngx_mail_close_connection(c);
+        return;
+    }
+
+    ngx_mail_init_session_handler(rev);
+}
+
+
+static void
 ngx_mail_init_session_handler(ngx_event_t *rev)
 {
     ngx_connection_t    *c;
@@ -242,9 +329,10 @@ ngx_mail_ssl_init_connection(ngx_ssl_t *
 
         s = c->data;
 
-        cscf = ngx_mail_get_module_srv_conf(s, ngx_mail_core_module);
-
-        ngx_add_timer(c->read, cscf->timeout);
+        if (!c->read->timer_set) {
+            cscf = ngx_mail_get_module_srv_conf(s, ngx_mail_core_module);
+            ngx_add_timer(c->read, cscf->timeout);
+        }
 
         c->ssl->handler = ngx_mail_ssl_handshake_handler;
 
# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1613793034 -10800
#      Sat Feb 20 06:50:34 2021 +0300
# Node ID 068d1bf4f2b3faaf84ec937c091274c940b5e1ca
# Parent  e7035246851a7c50ac7083f563ba754cc3ddb2f0
Mail: realip module.

When configured with the "set_real_ip_from", it can set client's IP
address as visible in logs to the one obtained via the PROXY protocol.

diff --git a/auto/modules b/auto/modules
--- a/auto/modules
+++ b/auto/modules
@@ -985,6 +985,12 @@ if [ $MAIL != NO ]; then
     ngx_module_srcs=src/mail/ngx_mail_proxy_module.c
 
     . auto/module
+
+    ngx_module_name=ngx_mail_realip_module
+    ngx_module_deps=
+    ngx_module_srcs=src/mail/ngx_mail_realip_module.c
+
+    . auto/module
 fi
 
 
diff --git a/src/mail/ngx_mail.h b/src/mail/ngx_mail.h
--- a/src/mail/ngx_mail.h
+++ b/src/mail/ngx_mail.h
@@ -408,6 +408,7 @@ char *ngx_mail_capabilities(ngx_conf_t *
 /* STUB */
 void ngx_mail_proxy_init(ngx_mail_session_t *s, ngx_addr_t *peer);
 void ngx_mail_auth_http_init(ngx_mail_session_t *s);
+ngx_int_t ngx_mail_realip_handler(ngx_mail_session_t *s);
 /**/
 
 
diff --git a/src/mail/ngx_mail_handler.c b/src/mail/ngx_mail_handler.c
--- a/src/mail/ngx_mail_handler.c
+++ b/src/mail/ngx_mail_handler.c
@@ -260,6 +260,11 @@ ngx_mail_proxy_protocol_handler(ngx_even
         return;
     }
 
+    if (ngx_mail_realip_handler(s) != NGX_OK) {
+        ngx_mail_close_connection(c);
+        return;
+    }
+
     ngx_mail_init_session_handler(rev);
 }
 
diff --git a/src/stream/ngx_stream_realip_module.c b/src/mail/ngx_mail_realip_module.c
copy from src/stream/ngx_stream_realip_module.c
copy to src/mail/ngx_mail_realip_module.c
--- a/src/stream/ngx_stream_realip_module.c
+++ b/src/mail/ngx_mail_realip_module.c
@@ -7,45 +7,29 @@
 
 #include <ngx_config.h>
 #include <ngx_core.h>
-#include <ngx_stream.h>
+#include <ngx_mail.h>
 
 
 typedef struct {
     ngx_array_t       *from;     /* array of ngx_cidr_t */
-} ngx_stream_realip_srv_conf_t;
-
-
-typedef struct {
-    struct sockaddr   *sockaddr;
-    socklen_t          socklen;
-    ngx_str_t          addr_text;
-} ngx_stream_realip_ctx_t;
+} ngx_mail_realip_srv_conf_t;
 
 
-static ngx_int_t ngx_stream_realip_handler(ngx_stream_session_t *s);
-static ngx_int_t ngx_stream_realip_set_addr(ngx_stream_session_t *s,
+static ngx_int_t ngx_mail_realip_set_addr(ngx_mail_session_t *s,
     ngx_addr_t *addr);
-static char *ngx_stream_realip_from(ngx_conf_t *cf, ngx_command_t *cmd,
+static char *ngx_mail_realip_from(ngx_conf_t *cf, ngx_command_t *cmd,
     void *conf);
-static void *ngx_stream_realip_create_srv_conf(ngx_conf_t *cf);
-static char *ngx_stream_realip_merge_srv_conf(ngx_conf_t *cf, void *parent,
+static void *ngx_mail_realip_create_srv_conf(ngx_conf_t *cf);
+static char *ngx_mail_realip_merge_srv_conf(ngx_conf_t *cf, void *parent,
     void *child);
-static ngx_int_t ngx_stream_realip_add_variables(ngx_conf_t *cf);
-static ngx_int_t ngx_stream_realip_init(ngx_conf_t *cf);
 
 
-static ngx_int_t ngx_stream_realip_remote_addr_variable(ngx_stream_session_t *s,
-    ngx_stream_variable_value_t *v, uintptr_t data);
-static ngx_int_t ngx_stream_realip_remote_port_variable(ngx_stream_session_t *s,
-    ngx_stream_variable_value_t *v, uintptr_t data);
-
-
-static ngx_command_t  ngx_stream_realip_commands[] = {
+static ngx_command_t  ngx_mail_realip_commands[] = {
 
     { ngx_string("set_real_ip_from"),
-      NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_TAKE1,
-      ngx_stream_realip_from,
-      NGX_STREAM_SRV_CONF_OFFSET,
+      NGX_MAIL_MAIN_CONF|NGX_MAIL_SRV_CONF|NGX_CONF_TAKE1,
+      ngx_mail_realip_from,
+      NGX_MAIL_SRV_CONF_OFFSET,
       0,
       NULL },
 
@@ -53,23 +37,22 @@ static ngx_command_t  ngx_stream_realip_
 };
 
 
-static ngx_stream_module_t  ngx_stream_realip_module_ctx = {
-    ngx_stream_realip_add_variables,       /* preconfiguration */
-    ngx_stream_realip_init,                /* postconfiguration */
+static ngx_mail_module_t  ngx_mail_realip_module_ctx = {
+    NULL,                                  /* protocol */
 
     NULL,                                  /* create main configuration */
     NULL,                                  /* init main configuration */
 
-    ngx_stream_realip_create_srv_conf,     /* create server configuration */
-    ngx_stream_realip_merge_srv_conf       /* merge server configuration */
+    ngx_mail_realip_create_srv_conf,       /* create server configuration */
+    ngx_mail_realip_merge_srv_conf         /* merge server configuration */
 };
 
 
-ngx_module_t  ngx_stream_realip_module = {
+ngx_module_t  ngx_mail_realip_module = {
     NGX_MODULE_V1,
-    &ngx_stream_realip_module_ctx,         /* module context */
-    ngx_stream_realip_commands,            /* module directives */
-    NGX_STREAM_MODULE,                     /* module type */
+    &ngx_mail_realip_module_ctx,           /* module context */
+    ngx_mail_realip_commands,              /* module directives */
+    NGX_MAIL_MODULE,                       /* module type */
     NULL,                                  /* init master */
     NULL,                                  /* init module */
     NULL,                                  /* init process */
@@ -81,70 +64,52 @@ ngx_module_t  ngx_stream_realip_module =
 };
 
 
-static ngx_stream_variable_t  ngx_stream_realip_vars[] = {
-
-    { ngx_string("realip_remote_addr"), NULL,
-      ngx_stream_realip_remote_addr_variable, 0, 0, 0 },
-
-    { ngx_string("realip_remote_port"), NULL,
-      ngx_stream_realip_remote_port_variable, 0, 0, 0 },
+ngx_int_t
+ngx_mail_realip_handler(ngx_mail_session_t *s)
+{
+    ngx_addr_t                   addr;
+    ngx_connection_t            *c;
+    ngx_mail_realip_srv_conf_t  *rscf;
 
-      ngx_stream_null_variable
-};
-
-
-static ngx_int_t
-ngx_stream_realip_handler(ngx_stream_session_t *s)
-{
-    ngx_addr_t                     addr;
-    ngx_connection_t              *c;
-    ngx_stream_realip_srv_conf_t  *rscf;
-
-    rscf = ngx_stream_get_module_srv_conf(s, ngx_stream_realip_module);
+    rscf = ngx_mail_get_module_srv_conf(s, ngx_mail_realip_module);
 
     if (rscf->from == NULL) {
-        return NGX_DECLINED;
+        return NGX_OK;
     }
 
     c = s->connection;
 
     if (c->proxy_protocol == NULL) {
-        return NGX_DECLINED;
+        return NGX_OK;
     }
 
     if (ngx_cidr_match(c->sockaddr, rscf->from) != NGX_OK) {
-        return NGX_DECLINED;
+        return NGX_OK;
     }
 
     if (ngx_parse_addr(c->pool, &addr, c->proxy_protocol->src_addr.data,
                        c->proxy_protocol->src_addr.len)
         != NGX_OK)
     {
-        return NGX_DECLINED;
+        return NGX_OK;
     }
 
     ngx_inet_set_port(addr.sockaddr, c->proxy_protocol->src_port);
 
-    return ngx_stream_realip_set_addr(s, &addr);
+    return ngx_mail_realip_set_addr(s, &addr);
 }
 
 
 static ngx_int_t
-ngx_stream_realip_set_addr(ngx_stream_session_t *s, ngx_addr_t *addr)
+ngx_mail_realip_set_addr(ngx_mail_session_t *s, ngx_addr_t *addr)
 {
-    size_t                    len;
-    u_char                   *p;
-    u_char                    text[NGX_SOCKADDR_STRLEN];
-    ngx_connection_t         *c;
-    ngx_stream_realip_ctx_t  *ctx;
+    size_t             len;
+    u_char            *p;
+    u_char             text[NGX_SOCKADDR_STRLEN];
+    ngx_connection_t  *c;
 
     c = s->connection;
 
-    ctx = ngx_palloc(c->pool, sizeof(ngx_stream_realip_ctx_t));
-    if (ctx == NULL) {
-        return NGX_ERROR;
-    }
-
     len = ngx_sock_ntop(addr->sockaddr, addr->socklen, text,
                         NGX_SOCKADDR_STRLEN, 0);
     if (len == 0) {
@@ -158,25 +123,19 @@ ngx_stream_realip_set_addr(ngx_stream_se
 
     ngx_memcpy(p, text, len);
 
-    ngx_stream_set_ctx(s, ctx, ngx_stream_realip_module);
-
-    ctx->sockaddr = c->sockaddr;
-    ctx->socklen = c->socklen;
-    ctx->addr_text = c->addr_text;
-
     c->sockaddr = addr->sockaddr;
     c->socklen = addr->socklen;
     c->addr_text.len = len;
     c->addr_text.data = p;
 
-    return NGX_DECLINED;
+    return NGX_OK;
 }
 
 
 static char *
-ngx_stream_realip_from(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
+ngx_mail_realip_from(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
 {
-    ngx_stream_realip_srv_conf_t *rscf = conf;
+    ngx_mail_realip_srv_conf_t *rscf = conf;
 
     ngx_int_t             rc;
     ngx_str_t            *value;
@@ -277,11 +236,11 @@ ngx_stream_realip_from(ngx_conf_t *cf, n
 
 
 static void *
-ngx_stream_realip_create_srv_conf(ngx_conf_t *cf)
+ngx_mail_realip_create_srv_conf(ngx_conf_t *cf)
 {
-    ngx_stream_realip_srv_conf_t  *conf;
+    ngx_mail_realip_srv_conf_t  *conf;
 
-    conf = ngx_pcalloc(cf->pool, sizeof(ngx_stream_realip_srv_conf_t));
+    conf = ngx_pcalloc(cf->pool, sizeof(ngx_mail_realip_srv_conf_t));
     if (conf == NULL) {
         return NULL;
     }
@@ -297,10 +256,10 @@ ngx_stream_realip_create_srv_conf(ngx_co
 
 
 static char *
-ngx_stream_realip_merge_srv_conf(ngx_conf_t *cf, void *parent, void *child)
+ngx_mail_realip_merge_srv_conf(ngx_conf_t *cf, void *parent, void *child)
 {
-    ngx_stream_realip_srv_conf_t *prev = parent;
-    ngx_stream_realip_srv_conf_t *conf = child;
+    ngx_mail_realip_srv_conf_t *prev = parent;
+    ngx_mail_realip_srv_conf_t *conf = child;
 
     if (conf->from == NULL) {
         conf->from = prev->from;
@@ -308,94 +267,3 @@ ngx_stream_realip_merge_srv_conf(ngx_con
 
     return NGX_CONF_OK;
 }
-
-
-static ngx_int_t
-ngx_stream_realip_add_variables(ngx_conf_t *cf)
-{
-    ngx_stream_variable_t  *var, *v;
-
-    for (v = ngx_stream_realip_vars; v->name.len; v++) {
-        var = ngx_stream_add_variable(cf, &v->name, v->flags);
-        if (var == NULL) {
-            return NGX_ERROR;
-        }
-
-        var->get_handler = v->get_handler;
-        var->data = v->data;
-    }
-
-    return NGX_OK;
-}
-
-
-static ngx_int_t
-ngx_stream_realip_init(ngx_conf_t *cf)
-{
-    ngx_stream_handler_pt        *h;
-    ngx_stream_core_main_conf_t  *cmcf;
-
-    cmcf = ngx_stream_conf_get_module_main_conf(cf, ngx_stream_core_module);
-
-    h = ngx_array_push(&cmcf->phases[NGX_STREAM_POST_ACCEPT_PHASE].handlers);
-    if (h == NULL) {
-        return NGX_ERROR;
-    }
-
-    *h = ngx_stream_realip_handler;
-
-    return NGX_OK;
-}
-
-
-static ngx_int_t
-ngx_stream_realip_remote_addr_variable(ngx_stream_session_t *s,
-    ngx_stream_variable_value_t *v, uintptr_t data)
-{
-    ngx_str_t                *addr_text;
-    ngx_stream_realip_ctx_t  *ctx;
-
-    ctx = ngx_stream_get_module_ctx(s, ngx_stream_realip_module);
-
-    addr_text = ctx ? &ctx->addr_text : &s->connection->addr_text;
-
-    v->len = addr_text->len;
-    v->valid = 1;
-    v->no_cacheable = 0;
-    v->not_found = 0;
-    v->data = addr_text->data;
-
-    return NGX_OK;
-}
-
-
-static ngx_int_t
-ngx_stream_realip_remote_port_variable(ngx_stream_session_t *s,
-    ngx_stream_variable_value_t *v, uintptr_t data)
-{
-    ngx_uint_t                port;
-    struct sockaddr          *sa;
-    ngx_stream_realip_ctx_t  *ctx;
-
-    ctx = ngx_stream_get_module_ctx(s, ngx_stream_realip_module);
-
-    sa = ctx ? ctx->sockaddr : s->connection->sockaddr;
-
-    v->len = 0;
-    v->valid = 1;
-    v->no_cacheable = 0;
-    v->not_found = 0;
-
-    v->data = ngx_pnalloc(s->connection->pool, sizeof("65535") - 1);
-    if (v->data == NULL) {
-        return NGX_ERROR;
-    }
-
-    port = ngx_inet_get_port(sa);
-
-    if (port > 0 && port < 65536) {
-        v->len = ngx_sprintf(v->data, "%ui", port) - v->data;
-    }
-
-    return NGX_OK;
-}
# HG changeset patch
# User Maxim Dounin <mdounin at mdounin.ru>
# Date 1613827679 -10800
#      Sat Feb 20 16:27:59 2021 +0300
# Node ID 30a66f88bb5d30bbfb445b57bd54c9ca35a345ec
# Parent  068d1bf4f2b3faaf84ec937c091274c940b5e1ca
Mail: sending of the PROXY protocol to backends.

Activated with the "proxy_protocol" directive.  Can be combined with
"listen ... proxy_protocol;" and "set_real_ip_from ...;" to pass
client address provided to nginx in the PROXY protocol header.

diff --git a/src/mail/ngx_mail.h b/src/mail/ngx_mail.h
--- a/src/mail/ngx_mail.h
+++ b/src/mail/ngx_mail.h
@@ -178,6 +178,7 @@ typedef enum {
 typedef struct {
     ngx_peer_connection_t   upstream;
     ngx_buf_t              *buffer;
+    ngx_uint_t              proxy_protocol;  /* unsigned  proxy_protocol:1; */
 } ngx_mail_proxy_ctx_t;
 
 
diff --git a/src/mail/ngx_mail_proxy_module.c b/src/mail/ngx_mail_proxy_module.c
--- a/src/mail/ngx_mail_proxy_module.c
+++ b/src/mail/ngx_mail_proxy_module.c
@@ -17,6 +17,7 @@ typedef struct {
     ngx_flag_t  pass_error_message;
     ngx_flag_t  xclient;
     ngx_flag_t  smtp_auth;
+    ngx_flag_t  proxy_protocol;
     size_t      buffer_size;
     ngx_msec_t  timeout;
 } ngx_mail_proxy_conf_t;
@@ -26,7 +27,8 @@ static void ngx_mail_proxy_block_read(ng
 static void ngx_mail_proxy_pop3_handler(ngx_event_t *rev);
 static void ngx_mail_proxy_imap_handler(ngx_event_t *rev);
 static void ngx_mail_proxy_smtp_handler(ngx_event_t *rev);
-static void ngx_mail_proxy_dummy_handler(ngx_event_t *ev);
+static void ngx_mail_proxy_write_handler(ngx_event_t *wev);
+static ngx_int_t ngx_mail_proxy_send_proxy_protocol(ngx_mail_session_t *s);
 static ngx_int_t ngx_mail_proxy_read_response(ngx_mail_session_t *s,
     ngx_uint_t state);
 static void ngx_mail_proxy_handler(ngx_event_t *ev);
@@ -82,6 +84,13 @@ static ngx_command_t  ngx_mail_proxy_com
       offsetof(ngx_mail_proxy_conf_t, smtp_auth),
       NULL },
 
+    { ngx_string("proxy_protocol"),
+      NGX_MAIL_MAIN_CONF|NGX_MAIL_SRV_CONF|NGX_CONF_FLAG,
+      ngx_conf_set_flag_slot,
+      NGX_MAIL_SRV_CONF_OFFSET,
+      offsetof(ngx_mail_proxy_conf_t, proxy_protocol),
+      NULL },
+
       ngx_null_command
 };
 
@@ -156,7 +165,7 @@ ngx_mail_proxy_init(ngx_mail_session_t *
     p->upstream.connection->pool = s->connection->pool;
 
     s->connection->read->handler = ngx_mail_proxy_block_read;
-    p->upstream.connection->write->handler = ngx_mail_proxy_dummy_handler;
+    p->upstream.connection->write->handler = ngx_mail_proxy_write_handler;
 
     pcf = ngx_mail_get_module_srv_conf(s, ngx_mail_proxy_module);
 
@@ -167,6 +176,8 @@ ngx_mail_proxy_init(ngx_mail_session_t *
         return;
     }
 
+    s->proxy->proxy_protocol = pcf->proxy_protocol;
+
     s->out.len = 0;
 
     switch (s->protocol) {
@@ -186,6 +197,12 @@ ngx_mail_proxy_init(ngx_mail_session_t *
         s->mail_state = ngx_smtp_start;
         break;
     }
+
+    if (rc == NGX_AGAIN) {
+        return;
+    }
+
+    ngx_mail_proxy_write_handler(p->upstream.connection->write);
 }
 
 
@@ -230,6 +247,17 @@ ngx_mail_proxy_pop3_handler(ngx_event_t 
         return;
     }
 
+    if (s->proxy->proxy_protocol) {
+        ngx_log_debug0(NGX_LOG_DEBUG_MAIL, c->log, 0, "mail proxy pop3 busy");
+
+        if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
+            ngx_mail_proxy_internal_server_error(s);
+            return;
+        }
+
+        return;
+    }
+
     rc = ngx_mail_proxy_read_response(s, 0);
 
     if (rc == NGX_AGAIN) {
@@ -353,6 +381,17 @@ ngx_mail_proxy_imap_handler(ngx_event_t 
         return;
     }
 
+    if (s->proxy->proxy_protocol) {
+        ngx_log_debug0(NGX_LOG_DEBUG_MAIL, c->log, 0, "mail proxy imap busy");
+
+        if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
+            ngx_mail_proxy_internal_server_error(s);
+            return;
+        }
+
+        return;
+    }
+
     rc = ngx_mail_proxy_read_response(s, s->mail_state);
 
     if (rc == NGX_AGAIN) {
@@ -499,6 +538,17 @@ ngx_mail_proxy_smtp_handler(ngx_event_t 
         return;
     }
 
+    if (s->proxy->proxy_protocol) {
+        ngx_log_debug0(NGX_LOG_DEBUG_MAIL, c->log, 0, "mail proxy smtp busy");
+
+        if (ngx_handle_read_event(c->read, 0) != NGX_OK) {
+            ngx_mail_proxy_internal_server_error(s);
+            return;
+        }
+
+        return;
+    }
+
     rc = ngx_mail_proxy_read_response(s, s->mail_state);
 
     if (rc == NGX_AGAIN) {
@@ -799,19 +849,92 @@ ngx_mail_proxy_smtp_handler(ngx_event_t 
 
 
 static void
-ngx_mail_proxy_dummy_handler(ngx_event_t *wev)
+ngx_mail_proxy_write_handler(ngx_event_t *wev)
 {
     ngx_connection_t    *c;
     ngx_mail_session_t  *s;
 
-    ngx_log_debug0(NGX_LOG_DEBUG_MAIL, wev->log, 0, "mail proxy dummy handler");
+    ngx_log_debug0(NGX_LOG_DEBUG_MAIL, wev->log, 0, "mail proxy write handler");
+
+    c = wev->data;
+    s = c->data;
+
+    if (s->proxy->proxy_protocol) {
+        if (ngx_mail_proxy_send_proxy_protocol(s) != NGX_OK) {
+            return;
+        }
+
+        s->proxy->proxy_protocol = 0;
+    }
 
     if (ngx_handle_write_event(wev, 0) != NGX_OK) {
-        c = wev->data;
-        s = c->data;
+        ngx_mail_proxy_internal_server_error(s);
+    }
+
+    if (c->read->ready) {
+        ngx_post_event(c->read, &ngx_posted_events);
+    }
+}
+
+
+static ngx_int_t
+ngx_mail_proxy_send_proxy_protocol(ngx_mail_session_t *s)
+{
+    u_char            *p;
+    ssize_t            n, size;
+    ngx_connection_t  *c;
+    u_char             buf[NGX_PROXY_PROTOCOL_MAX_HEADER];
+
+    s->connection->log->action = "sending PROXY protocol header to upstream";
+
+    ngx_log_debug0(NGX_LOG_DEBUG_MAIL, s->connection->log, 0,
+                   "mail proxy send PROXY protocol header");
+
+    p = ngx_proxy_protocol_write(s->connection, buf,
+                                 buf + NGX_PROXY_PROTOCOL_MAX_HEADER);
+    if (p == NULL) {
+        ngx_mail_proxy_internal_server_error(s);
+        return NGX_ERROR;
+    }
+
+    c = s->proxy->upstream.connection;
+
+    size = p - buf;
 
-        ngx_mail_proxy_close_session(s);
+    n = c->send(c, buf, size);
+
+    if (n == NGX_AGAIN) {
+        if (ngx_handle_write_event(c->write, 0) != NGX_OK) {
+            ngx_mail_proxy_internal_server_error(s);
+            return NGX_ERROR;
+        }
+
+        return NGX_AGAIN;
+    }
+
+    if (n == NGX_ERROR) {
+        ngx_mail_proxy_internal_server_error(s);
+        return NGX_ERROR;
     }
+
+    if (n != size) {
+
+        /*
+         * PROXY protocol specification:
+         * The sender must always ensure that the header
+         * is sent at once, so that the transport layer
+         * maintains atomicity along the path to the receiver.
+         */
+
+        ngx_log_error(NGX_LOG_ERR, s->connection->log, 0,
+                      "could not send PROXY protocol header at once");
+
+        ngx_mail_proxy_internal_server_error(s);
+
+        return NGX_ERROR;
+    }
+
+    return NGX_OK;
 }
 
 
@@ -1212,6 +1335,7 @@ ngx_mail_proxy_create_conf(ngx_conf_t *c
     pcf->pass_error_message = NGX_CONF_UNSET;
     pcf->xclient = NGX_CONF_UNSET;
     pcf->smtp_auth = NGX_CONF_UNSET;
+    pcf->proxy_protocol = NGX_CONF_UNSET;
     pcf->buffer_size = NGX_CONF_UNSET_SIZE;
     pcf->timeout = NGX_CONF_UNSET_MSEC;
 
@@ -1229,6 +1353,7 @@ ngx_mail_proxy_merge_conf(ngx_conf_t *cf
     ngx_conf_merge_value(conf->pass_error_message, prev->pass_error_message, 0);
     ngx_conf_merge_value(conf->xclient, prev->xclient, 1);
     ngx_conf_merge_value(conf->smtp_auth, prev->smtp_auth, 0);
+    ngx_conf_merge_value(conf->proxy_protocol, prev->proxy_protocol, 0);
     ngx_conf_merge_size_value(conf->buffer_size, prev->buffer_size,
                               (size_t) ngx_pagesize);
     ngx_conf_merge_msec_value(conf->timeout, prev->timeout, 24 * 60 * 60000);

-- 
Maxim Dounin
http://mdounin.ru/


More information about the nginx-devel mailing list