NGINX stripping websocket "Upgrade" headers

[bengalih]

I have the following in my site.conf file:

------------------------------------------------------------
        #Websockets
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
------------------------------------------------------------

My understanding is that this will set the proper headers for websocket
communications.

Specifically, it will add a "Connection" header with value "Upgrade" and add
an "Upgrade" header with whatever value the client passes to it in the
"Upgrade" header (i.e. add the header on for hop-to-hop to upstream server).
In the case of websockets, the Upgrade header will be "websocket."

I am executing the following curl command to my server directly on my LAN
(no NGINX involved)

------------------------------------------------------------
curl -k --include --no-buffer --header "Connection: Upgrade" --header
"Upgrade: websocket" --header "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ=="
--header "Sec-WebSocket-Version: 13" --header "X-Plex-Token:
87LJshwQwRxT24jaY8Be" https://10.10.10.102:32400/:/websockets/notifications
------------------------------------------------------------

This results in the following headers being passed to the server (via packet
capture on the server):

------------------------------------------------------------
Frame 1268: 307 bytes on wire (2456 bits), 307 bytes captured (2456 bits) on
interface 0
Ethernet II, Src: IntelCor_2f:f6:e4 (24:77:03:2f:f6:e4), Dst:
AsrockIn_7c:fe:cf (70:85:c2:7c:fe:cf)
Internet Protocol Version 4, Src: 10.10.10.201, Dst: 10.10.10.102
Transmission Control Protocol, Src Port: 53019, Dst Port: 32400, Seq: 1,
Ack: 1, Len: 253
Hypertext Transfer Protocol
    GET /:/websockets/notifications HTTP/1.1\r\n
    Host: 10.10.10.102:32400\r\n
    User-Agent: curl/7.68.0\r\n
    Accept: */*\r\n
    Connection: Upgrade\r\n
    Upgrade: websocket\r\n
    Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==\r\n
    Sec-WebSocket-Version: 13\r\n
    X-Plex-Token: 87LJshwQwRxT24jaY8Be\r\n
    \r\n
    [Full request URI:
http://10.10.10.102:32400/:/websockets/notifications]
    [HTTP request 1/1]
    [Response in frame: 1269]
------------------------------------------------------------

This appears as normal, and is met with a proper "Switching Protocols"
result.

However, when I send this request out to my public IP through NGINX, my
server sees this in the headers:

------------------------------------------------------------
Frame 2494: 381 bytes on wire (3048 bits), 381 bytes captured (3048 bits) on
interface 0
Ethernet II, Src: AsustekC_c6:4c:30 (1c:b7:2c:c6:4c:30), Dst:
AsrockIn_7c:fe:cf (70:85:c2:7c:fe:cf)
Internet Protocol Version 4, Src: 10.10.10.1, Dst: 10.10.10.102
Transmission Control Protocol, Src Port: 40461, Dst Port: 32400, Seq: 1,
Ack: 1, Len: 315
Hypertext Transfer Protocol
    GET /:/websockets/notifications HTTP/1.1\r\n
    Host: plex.mydomain.com\r\n
    X-Real-IP: 10.10.10.201\r\n
    X-Forwarded-For: 10.10.10.201\r\n
    X-Forwarded-Proto: https\r\n
    Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==\r\n
    Sec-WebSocket-Version: 13\r\n
    Connection: Upgrade\r\n
    user-agent: curl/7.68.0\r\n
    accept: */*\r\n
    x-plex-token: 87LJshwQwRxT24jaY8Be\r\n
    \r\n
    [Full request URI: http://plex.mydomain.com/:/websockets/notifications]
    [HTTP request 1/1]
------------------------------------------------------------

Note that while the "Connection" header has been added (presumably because
it is added specifically in the conf file via text and not passed header
variable), the "Upgrade" header is missing.

If I change my /conf file to look like this:

------------------------------------------------------------
        #Websockets
        proxy_http_version 1.1;
        proxy_set_header Upgrade websocket;
        proxy_set_header Connection "Upgrade";
------------------------------------------------------------

Then the header is properly added.

My understanding is NGINX strips/doesn't pass along any empty headers. Based
on my my local IP test, it is clear that the appropriate header is set (you
can see it in my sent curl statement and in the packet that the server
receives). However, due to the fact that it is not passed on to the upstream
server via NGINX, it would appear that NGINX thinks the $http_upgrade is
empty from the client and therefore not passing it on.

Can anyone explain this?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,286719,286719#msg-286719