IP Transparency in NGINX

[jbalasubramanian]

Hi all,

I am using NGINX 1.13.5 as a Load Balancer for one of my CUSTOM-APPLICATION
which will listen on UDP port 2231,67 and 68. 

I am trying for Load Balancing with IP-Transparency.


When I using the proxy_protocol method the packets received from a remote
client is modified and send to upstream by NGINX LB not sure why/how the
packet is modified and also the remote client IP is NOT as source IP.     

When I using proxy_bind, the packet is forwarded to configured upstream but
the source IP is not updated with Remote Client IP.

Basically, in both methods, the remote client address was not used as a
source IP. I hope I missed some minor parts. Can someone help to resolve
this issue?

 The following are the detailed configuration for your reference.

Method 1 :- proxy_protocol

Configuration:

user  root;
worker_processes  1;
error_log  /var/log/nginx/error.log debug;
pid        /var/run/nginx.pid;
events {
    worker_connections  1024;
}
stream {
    server {
        listen 10.43.18.107:2231 udp;
        proxy_protocol on;
        proxy_pass 10.43.18.172:2231;
    }
    server {
        listen 10.43.18.107:67 udp;
        proxy_protocol on;
        proxy_pass 10.43.18.172:67;
    }
    server {
        listen 10.43.18.107:68 udp;
        proxy_protocol on;
        proxy_pass 10.43.18.172:68;
    }
}

TCPDUMP O/P :

>From LB:

10:05:07.284259 IP 10.43.18.116.2231 > 10.43.18.107.2231: UDP, length 43
10:05:07.284555 IP 10.43.18.107.51775 > 10.43.18.172.2231: UDP, length 91

>From upstream[Custom application]:

10:05:07.284442 IP 10.43.18.107.51775 > 10.43.18.172.2231: UDP, length 91

Method 2:- [ proxy_bind ]

Configuration:

user  root;
worker_processes  1;
error_log  /var/log/nginx/error.log debug;
pid        /var/run/nginx.pid;
events {
    worker_connections  1024;
}

stream {
    server {
        listen 10.43.18.107:2231 udp;
        proxy_bind $remote_addr:2231 transparent;
        proxy_pass 10.43.18.172:2231;
    }
    server {
        listen 10.43.18.107:67 udp;
        proxy_bind $remote_addr:67 transparent;
        proxy_pass 10.43.18.172:67;
    }
    server {
        listen 10.43.18.107:68 udp;
        proxy_bind $remote_addr:68 transparent;
        proxy_pass 10.43.18.172:68;
    }

}

Also, added the below rules :

ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100
iptables -t mangle -A PREROUTING -p udp -s 10.43.18.0/24 --sport 2231 -j
MARK --set-xmark 0x1/0xffffffff
iptables -t mangle -A PREROUTING -p udp -s 10.43.18.0/24 --sport 67 -j MARK
--set-xmark 0x1/0xffffffff
iptables -t mangle -A PREROUTING -p udp -s 10.43.18.0/24 --sport 68 -j MARK
--set-xmark 0x1/0xffffffff

However, still, the packet is sent from NGINX LB with its own IP, not with
the remote client IP address.

TCPDUMP O/P from LB:

11:49:51.999829 IP 10.43.18.116.2231 > 10.43.18.107.2231: UDP, length 43
11:49:52.000161 IP 10.43.18.107.2231 > 10.43.18.172.2231: UDP, length 43

TPCDUM O/P from Upstream:
 
11:49:52.001155 IP 10.43.18.107.2231 > 10.43.18.172.2231: UDP, length 43


Note: I have followed the below link.  

https://www.nginx.com/blog/ip-transparency-direct-server-return-nginx-plus-transparent-proxy/

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,284810,284810#msg-284810