How to configure Nginx LB IP-Transparency for custom UDP application

[Roman Arutyunyan]

Hi,

On Tue, Jul 09, 2019 at 05:55:39PM +0530, Jeya Murugan wrote:
> Hi all,
> 
> 
> I am using *NGINX 1.13.5 as a Load Balancer for one of my
> CUSTOM-APPLICATION *which will listen on* UDP port 2231,67 and 68.*
> 
> I am trying for Load Balancing with IP-Transparency.
> 
> 
> 
> When I using the proxy_protocol method the packets received from a remote
> client is modified and send to upstream by NGINX LB not sure why/how the
> packet is modified and also the remote client IP is NOT as source IP.

The proxy_protocol directive adds a PROXY protocol header to the datagram,
that's why it's modified.  The directive does not change the source address.
Instead, the remote client address is passed in the PROXY protocol header.

> When I using proxy_bind, the packet is forwarded to configured upstream but
> the source IP is not updated with Remote Client IP.

What is the reason for the port next to $remote_addr in proxy_bind?
Also make sure nginx master runs with sufficient privileges.

> *Basically, in both methods, the remote client address was not used as a
> source IP. I hope I missed some minor parts. Can someone help to resolve
> this issue?*
> 
> 
> 
> The following are the detailed configuration for your reference.
> 
> 
> 
> *Method 1 :- proxy_protocol*
> 
> 
> 
> *Configuration:*
> 
> 
> 
> user  *root;*
> worker_processes  1;
> error_log  /var/log/nginx/error.log debug;
> pid        /var/run/nginx.pid;
> events {
>     worker_connections  1024;
> 
> }
> 
> stream {
>     server {
>         listen 10.43.18.107:2231 udp;
>         proxy_protocol on;
>         proxy_pass 10.43.18.172:2231;
>     }
>     server {
>         listen 10.43.18.107:67 udp;
>         proxy_protocol on;
>         proxy_pass 10.43.18.172:67;
>     }
>     server {
>         listen 10.43.18.107:68 udp;
>         proxy_protocol on;
>         proxy_pass 10.43.18.172:68;
>     }
> }
> 
> *TCPDUMP O/P :*
> 
> 
> 
> *From LB:*
> 
> 10:05:07.284259 IP 10.43.18.116.2231 > 10.43.18.107.2231: UDP, length 43
> 
> 10:05:07.284555 IP 10.43.18.107.51775 > 10.43.18.172.2231: UDP, length 91
> 
> 
> 
> *From upstream[Custom application]:*
> 
> 10:05:07.284442 IP 10.43.18.107.51775 > 10.43.18.172.2231: UDP, length 91
> 
> 
> 
> *Method 2:- [ proxy_bind ]*
> 
> 
> 
> *Configuration:*
> 
> 
> 
> user  root;
> worker_processes  1;
> error_log  /var/log/nginx/error.log debug;
> pid        /var/run/nginx.pid;
> events {
>     worker_connections  1024;
> }
> 
> stream {
>     server {
>         listen 10.43.18.107:2231 udp;
>         proxy_bind $remote_addr:2231 transparent;
>         proxy_pass 10.43.18.172:2231;
>     }
>     server {
>         listen 10.43.18.107:67 udp;
>         proxy_bind $remote_addr:67 transparent;
>         proxy_pass 10.43.18.172:67;
>     }
>     server {
>         listen 10.43.18.107:68 udp;
>         proxy_bind $remote_addr:68 transparent;
>         proxy_pass 10.43.18.172:68;
>     }
> 
> }
> 
> 
> 
> *Also, added the below rules :*
> 
> 
> 
> ip rule add fwmark 1 lookup 100
> 
> ip route add local 0.0.0.0/0 dev lo table 100
> iptables -t mangle -A PREROUTING -p udp -s 10.43.18.0/24 --sport 2231 -j
> MARK --set-xmark 0x1/0xffffffff
> iptables -t mangle -A PREROUTING -p udp -s 10.43.18.0/24 --sport 67 -j MARK
> --set-xmark 0x1/0xffffffff
> iptables -t mangle -A PREROUTING -p udp -s 10.43.18.0/24 --sport 68 -j MARK
> --set-xmark 0x1/0xffffffff
> 
> 
> 
> However, still, the packet is sent from NGINX LB with its own IP, not with
> the remote client IP address.
> 
> 
> 
> *TCPDUMP O/P from LB:*
> 
> 
> 
> 11:49:51.999829 IP 10.43.18.116.2231 > 10.43.18.107.2231: UDP, length 43
> 
> 11:49:52.000161 IP 10.43.18.107.2231 > 10.43.18.172.2231: UDP, length 43
> 
> 
> 
> *TPCDUM O/P from Upstream:*
> 
> 
> 
> 11:49:52.001155 IP 10.43.18.107.2231 > 10.43.18.172.2231: UDP, length 43
> 
> 
> 
> *Note:* I have followed the below link.
> 
> 
> 
> https://www.nginx.com/blog/ip-transparency-direct-server-return-nginx-plus-transparent-proxy/

> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx


-- 
Roman Arutyunyan