How to configure Nginx LB IP-Transparency for custom UDP application

[Jeya Murugan]

>
> @all : Can someone help /point-out what i have missed in proxy_protocol
>> here?
>>
>> > I am using *NGINX 1.13.5 as a Load Balancer for one of my
>> > CUSTOM-APPLICATION *which will listen on* UDP port 2231,67 and 68.*
>> >
>> > I am trying for Load Balancing with IP-Transparency.
>> >
>> >
>> >
>> > When I using the proxy_protocol method the packets received from a
>> remote
>> > client is modified and send to upstream by NGINX LB not sure why/how the
>> > packet is modified and also the remote client IP is NOT as source IP.
>>
>> The proxy_protocol directive adds a PROXY protocol header to the datagram,
>> that's why it's modified.  The directive does not change the source
>> address.
>> Instead, the remote client address is passed in the PROXY protocol header.
>>
>> <jeya> : Okay. Do we have any options to send remote client IP as source
>> address? Due to additional proxy header the packet is dropped by the
>> application running in the upstream. How can the proxy header can be
>> stripped in the upstream end?
>
>                     Do we need to do configuration/rules on the upstream
> end?
>
>
>> > When I using proxy_bind, the packet is forwarded to configured upstream
>> but
>> > the source IP is not updated with Remote Client IP.
>>
>> What is the reason for the port next to $remote_addr in proxy_bind?
>> Also make sure nginx master runs with sufficient privileges.
>>
>
> <Jeya> : Yes, application running with root privilege as specified in the
> conf file
>
> Also, the proxy_bind syntax is referred in the below link.'
>
> https://www.nginx.com/blog/ip-transparency-direct-server-return-nginx-plus-transparent-proxy/#proxy_bind
>
> proxy_bind $remote_addr:$remote_port transparent;
>
>>
>> > *Basically, in both methods, the remote client address was not used as a
>> > source IP. I hope I missed some minor parts. Can someone help to resolve
>> > this issue?*
>> >
>> >
>> >
>> > The following are the detailed configuration for your reference.
>> >
>> >
>> >
>> > *Method 1 :- proxy_protocol*
>> >
>> >
>> >
>> > *Configuration:*
>> >
>> >
>> >
>> > user  *root;*
>> > worker_processes  1;
>> > error_log  /var/log/nginx/error.log debug;
>> > pid        /var/run/nginx.pid;
>> > events {
>> >     worker_connections  1024;
>> >
>> > }
>> >
>> > stream {
>> >     server {
>> >         listen 10.43.18.107:2231 udp;
>> >         proxy_protocol on;
>> >         proxy_pass 10.43.18.172:2231;
>> >     }
>> >     server {
>> >         listen 10.43.18.107:67 udp;
>> >         proxy_protocol on;
>> >         proxy_pass 10.43.18.172:67;
>> >     }
>> >     server {
>> >         listen 10.43.18.107:68 udp;
>> >         proxy_protocol on;
>> >         proxy_pass 10.43.18.172:68;
>> >     }
>> > }
>> >
>> > *TCPDUMP O/P :*
>> >
>> >
>> >
>> > *From LB:*
>> >
>> > 10:05:07.284259 IP 10.43.18.116.2231 > 10.43.18.107.2231: UDP, length 43
>> >
>> > 10:05:07.284555 IP 10.43.18.107.51775 > 10.43.18.172.2231: UDP, length
>> 91
>> >
>> >
>> >
>> > *From upstream[Custom application]:*
>> >
>> > 10:05:07.284442 IP 10.43.18.107.51775 > 10.43.18.172.2231: UDP, length
>> 91
>> >
>> >
>> >
>> > *Method 2:- [ proxy_bind ]*
>> >
>> >
>> >
>> > *Configuration:*
>> >
>> >
>> >
>> > user  root;
>> > worker_processes  1;
>> > error_log  /var/log/nginx/error.log debug;
>> > pid        /var/run/nginx.pid;
>> > events {
>> >     worker_connections  1024;
>> > }
>> >
>> > stream {
>> >     server {
>> >         listen 10.43.18.107:2231 udp;
>> >         proxy_bind $remote_addr:2231 transparent;
>> >         proxy_pass 10.43.18.172:2231;
>> >     }
>> >     server {
>> >         listen 10.43.18.107:67 udp;
>> >         proxy_bind $remote_addr:67 transparent;
>> >         proxy_pass 10.43.18.172:67;
>> >     }
>> >     server {
>> >         listen 10.43.18.107:68 udp;
>> >         proxy_bind $remote_addr:68 transparent;
>> >         proxy_pass 10.43.18.172:68;
>> >     }
>> >
>> > }
>> >
>> >
>> >
>> > *Also, added the below rules :*
>> >
>> >
>> >
>> > ip rule add fwmark 1 lookup 100
>> >
>> > ip route add local 0.0.0.0/0 dev lo table 100
>> > iptables -t mangle -A PREROUTING -p udp -s 10.43.18.0/24 --sport 2231
>> -j
>> > MARK --set-xmark 0x1/0xffffffff
>> > iptables -t mangle -A PREROUTING -p udp -s 10.43.18.0/24 --sport 67 -j
>> MARK
>> > --set-xmark 0x1/0xffffffff
>> > iptables -t mangle -A PREROUTING -p udp -s 10.43.18.0/24 --sport 68 -j
>> MARK
>> > --set-xmark 0x1/0xffffffff
>> >
>> >
>> >
>> > However, still, the packet is sent from NGINX LB with its own IP, not
>> with
>> > the remote client IP address.
>> >
>> >
>> >
>> > *TCPDUMP O/P from LB:*
>> >
>> >
>> >
>> > 11:49:51.999829 IP 10.43.18.116.2231 > 10.43.18.107.2231: UDP, length 43
>> >
>> > 11:49:52.000161 IP 10.43.18.107.2231 > 10.43.18.172.2231: UDP, length 43
>> >
>> >
>> >
>> > *TPCDUM O/P from Upstream:*
>> >
>> >
>> >
>> > 11:49:52.001155 IP 10.43.18.107.2231 > 10.43.18.172.2231: UDP, length 43
>> >
>> >
>> >
>> > *Note:* I have followed the below link.
>> >
>> >
>> >
>> >
>> https://www.nginx.com/blog/ip-transparency-direct-server-return-nginx-plus-transparent-proxy/
>>
>> > _______________________________________________
>> > nginx mailing list
>> > nginx at nginx.org
>> > http://mailman.nginx.org/mailman/listinfo/nginx
>>
>>
>> --
>> Roman Arutyunyan
>> _______________________________________________
>> nginx mailing list
>> nginx at nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20190716/679a3a60/attachment-0001.html>