nginx behind load balancer

[Rami Essaid]

As a quick update, it looks like this has happened before. The load
balancer bounces off of several internal IP's sometimes and nginx picks the
last one only. Does anyone know of a workaround to remove the last two
trusted IPs from the x-forwarded-for header?

http://forum.nginx.org/read.php?11,26102,214069

Rami

On Fri, Nov 25, 2011 at 4:54 PM, Rami Essaid <rami.essaid at gmail.com> wrote:

> In looking at the $proxy_add_x_forwarded_for variable I believe that the
> Loadbalancer is in fact passing the variables but nginx is taking the wrong
> value?
>
> Here is what i get from the variables.
> $proxy_add_x_forwarded_for: 217.27.244.18, 10.160.43.200, 10.160.43.200
> $remote_addr: 10.160.43.200
>
> Does this mean that nginx is taking the last value in the x-forwarded-for?
>
>
> To answer your other question, yes, the Amazon plenty of sockets
> available to connect to nginx.
>
> The reason we added this was to be able to utilize Amazon's auto scaling
> feature for redundancy and scalability.  We have not been able to test
> performance yet since we are still trying to make it work but as soon as we
> do I will be sure to report back.
>
>
> On Fri, Nov 25, 2011 at 4:19 PM, Stefan Caunter <stef at caunter.ca> wrote:
>
>> On Fri, Nov 25, 2011 at 12:37 PM, Rami Essaid <rami.essaid at gmail.com>
>> wrote:
>> > Thanks for the help Maxim!  We disabled our limit_req and that seemed to
>> > have fixed the problem.  Looking at the logs it seems that only 1/3 of
>> the
>> > requests are correctly getting the new IP assigned via the realIP
>> module,
>> > the remainder are still logging the load balancer IP.  This probably is
>> more
>> > of an issue with the amazon load balancer but do you have any idea on
>> what
>> > may be going on?
>> > Also, where would you recommend as a place to start tracking and fixing
>> the
>> > other issue?
>> >
>>
>> You have not established that load balancer is setting X-Forwarded-For
>> or some other header to pass Real IP.
>>
>> Does the amazon pseudo device have enough sockets available to connect to
>> nginx?
>>
>> What do you expect from adding this layer? Were you hitting
>> performance limits? Has it improved or degraded performance?
>>
>> >
>> >
>> > On Fri, Nov 25, 2011 at 12:14 PM, Maxim Dounin <mdounin at mdounin.ru>
>> wrote:
>> >>
>> >> Hello!
>> >>
>> >> On Fri, Nov 25, 2011 at 09:54:14AM -0500, Rami Essaid wrote:
>> >>
>> >> > Hi Maxim,
>> >> >
>> >> > We implemented to module and still had some trouble.  A lot of the
>> >> > connections would return " 503 Service Temporarily Unavailable".  Our
>> >> > configuration works fine without the load balancer but then gives
>> these
>> >> > 503
>> >> > errors behind the load balancer.
>> >>
>> >> nginx itself will only return 503 if it hits either limit_conn or
>> >> limit_req.
>> >>
>> >> If you see this returned by nginx, and it only happens with load
>> >> balancer, this may indicate you've not configured realip module
>> >> properly (or your load balancer doesn't provide appropriate
>> >> headers) and you are hitting per-ip limits configured due to all
>> >> requests appear to be from load balancer.
>> >>
>> >> Check if client's ip logged is really client's one, not an ip of
>> >> your load balancer.
>> >>
>> >> > Looking into the error logs I notice a lot of these errors both with
>> and
>> >> > without the load balancer "connect() failed (111: Connection refused)
>> >> > while
>> >> > connecting to upstream".  Could this be the reason that we are having
>> >> > issues?
>> >>
>> >> Unlikely, but It's a good idea to track and fix this in any case.
>> >>
>> >> Maxim Dounin
>> >>
>> >> >
>> >> > Thanks!
>> >> > Rami
>> >> >
>> >> >
>> >> > On Mon, Nov 21, 2011 at 7:46 AM, Maxim Dounin <mdounin at mdounin.ru>
>> >> > wrote:
>> >> >
>> >> > > Hello!
>> >> > >
>> >> > > On Mon, Nov 21, 2011 at 07:40:49AM -0500, Rami Essaid wrote:
>> >> > >
>> >> > > > Thanks Maxim,
>> >> > > >
>> >> > > > This looks like exactly what we need.  In your experience does
>> this
>> >> > > resolve
>> >> > > > most issues behind a load balancer?
>> >> > >
>> >> > > Yes.
>> >> > >
>> >> > > Maxim Dounin
>> >> > >
>> >> > > > On Mon, Nov 21, 2011 at 7:38 AM, Maxim Dounin <
>> mdounin at mdounin.ru>
>> >> > > wrote:
>> >> > > >
>> >> > > > > Hello!
>> >> > > > >
>> >> > > > > On Mon, Nov 21, 2011 at 07:25:39AM -0500, Rami Essaid wrote:
>> >> > > > >
>> >> > > > > > Hi Guys,
>> >> > > > > >
>> >> > > > > > This weekend for scalability we tried putting our nginx
>> servers
>> >> > > behind
>> >> > > > > > amazon's elastic load balancers and came across a road
>> block: it
>> >> > > does not
>> >> > > > > > transparently pass the user IP and header information to
>> nginx.
>> >> > > > > > This
>> >> > > > > caused
>> >> > > > > > issues with several pieces of nginx we use including the IP
>> >> > > > > > allow /
>> >> > > deny
>> >> > > > > > rules, the limit_req module, and the limit_con module.  Has
>> >> > > > > > anyone
>> >> > > > > > successfully put nginx behind a load balancer?  Any ideas on
>> how
>> >> > > > > > to
>> >> > > make
>> >> > > > > > this work?
>> >> > > > >
>> >> > > > > http://wiki.nginx.org/HttpRealIpModule
>> >> > > > >
>> >> > > > > Maxim Dounin
>> >> > > > >
>> >> > > > > _______________________________________________
>> >> > > > > nginx mailing list
>> >> > > > > nginx at nginx.org
>> >> > > > > http://mailman.nginx.org/mailman/listinfo/nginx
>> >> > > > >
>> >> > >
>> >> > > > _______________________________________________
>> >> > > > nginx mailing list
>> >> > > > nginx at nginx.org
>> >> > > > http://mailman.nginx.org/mailman/listinfo/nginx
>> >> > >
>> >> > > _______________________________________________
>> >> > > nginx mailing list
>> >> > > nginx at nginx.org
>> >> > > http://mailman.nginx.org/mailman/listinfo/nginx
>> >> > >
>> >>
>> >> > _______________________________________________
>> >> > nginx mailing list
>> >> > nginx at nginx.org
>> >> > http://mailman.nginx.org/mailman/listinfo/nginx
>> >>
>> >> _______________________________________________
>> >> nginx mailing list
>> >> nginx at nginx.org
>> >> http://mailman.nginx.org/mailman/listinfo/nginx
>> >
>> >
>> > _______________________________________________
>> > nginx mailing list
>> > nginx at nginx.org
>> > http://mailman.nginx.org/mailman/listinfo/nginx
>> >
>>
>> _______________________________________________
>> nginx mailing list
>> nginx at nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20111125/b8e44921/attachment-0001.html>