[Solved] IPv6 connectivity issue to `nginx.org` due to tunnel MTU

[Alexander Huynh]

Quick follow-up: after more digging I've found that I can solve it another way: using router advertisements to set the MTU.

Here's a tcpdump showing the problem, note how the MSS is 1440 bidirectionally:

    IP6 (flowlabel 0x47dd6, hlim 64, next-header TCP (6) payload length: 44) 2001:db8::1.55652 > 2a05:d014:edb:5704::6.443: Flags [S], cksum 0xffc3 (correct), seq 3046951728, win 65535, options [mss 1440,nop,wscale 6,nop,nop,TS val 1569227211 ecr 0,sackOK,eol], length 0
    IP6 (flowlabel 0x68d1d, hlim 41, next-header TCP (6) payload length: 40) 2a05:d014:edb:5704::6.443 > 2001:db8::1.55652: Flags [S.], cksum 0x1350 (correct), seq 2416054786, ack 3046951729, win 8192, options [mss 1440,sackOK,TS val 3541526609 ecr 1569227211,nop,wscale 0], length 0

And here's using MSS clamping to have the router rewrite MSS fields it sees, resulting in a smaller return MSS:

    IP6 (flowlabel 0xf55e4, hlim 64, next-header TCP (6) payload length: 44) 2001:db8::1.55198 > 2a05:d014:edb:5704::6.443: Flags [S], cksum 0xe188 (correct), seq 2163962289, win 65535, options [mss 1440,nop,wscale 6,nop,nop,TS val 1569009648 ecr 0,sackOK,eol], length 0
    IP6 (flowlabel 0x18b17, hlim 41, next-header TCP (6) payload length: 40) 2a05:d014:edb:5704::6.443 > 2001:db8::1.55198: Flags [S.], cksum 0x3197 (correct), seq 3274704991, ack 2163962290, win 8192, options [mss 1420,sackOK,TS val 3541303899 ecr 1569009648,nop,wscale 0], length 0

By instead using router advertisements, a co-operating OS doesn't have to rely on the router rewrites, and can just do the right thing™ from the start:

    IP6 (flowlabel 0x8ce90, hlim 64, next-header TCP (6) payload length: 44) 2001:db8::1.55859 > 2a05:d014:edb:5704::6.443: Flags [S], cksum 0xf73a (correct), seq 3178375029, win 65535, options [mss 1420,nop,wscale 6,nop,nop,TS val 1569335937 ecr 0,sackOK,eol], length 0
    IP6 (flowlabel 0x8e947, hlim 41, next-header TCP (6) payload length: 40) 2a05:d014:edb:5704::6.443 > 2001:db8::1.55859: Flags [S.], cksum 0x314c (correct), seq 2240678375, ack 3178375030, win 8192, options [mss 1440,sackOK,TS val 2237309444 ecr 1569335937,nop,wscale 0], length 0

Note the initial lower MSS of 1420.

Given the two options, I think using router advertisements is better than MSS clamping, for multiple reasons:

  * the problem packets aren't even generated at all, vs. being constructed and rewritten
  * MTU applies to more than just TCP, resolving potential UDP / ICMP / etc. issues
  * RAs are more granular, and is more intuitive to apply to different interfaces, including VLANs
  * fewer numbers need to be memorized, i.e. translating MTU to MSS

If you have an EdgeRouter and wish to use RAs, then replace the following MSS clamping commands:

>  * set firewall options mss-clamp6 mss 1420
>  * set firewall options mss-clamp6 interface-type all


With the following RA MTU commands:

  * set interfaces switch switch0 ipv6 router-advert link-mtu 1480

Replace `switch switch0` with `ethernet ethX` for an ethernet interface.

Thanks again for reading, and I hope this helps someone down the line.
-- 
Alex