Performance test caps at 600 Mbit/s

[CJ Ess]

Out of the box Linux isn't optimized for high speed networking - if you
google for linux and 1g or 10g you'll find a ton of pages about
configuration changes.

I think you'll want to do something like:

ifconfig eth0 txqueuelen 10000
ethtool -G eth0 rx 4096 tx 4096
ip route | while read x; do ip route change $x initcwnd 10 initrwnd 10; done

And these sysctl settings:
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_probes = 14
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.tcp_sack = 1
net.ipv4.ip_local_port_range = 1024 65535
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.core.rmem_default = 524288
net.core.rmem_default = 524288
net.ipv4.tcp_rmem = 4096 524288 33554432
net.ipv4.tcp_wmem = 4096 524288 33554432
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_max_syn_backlog = 65536
net.ipv4.tcp_max_orphans = 262144
net.core.netdev_max_backlog = 300000
net.core.somaxconn = 65536
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_slow_start_after_idle = 0

At least these are the settings I've used for benchmarking, may do less or
more on your setup. The NIC vendor can be really important - Intel NICs are
really good and work really well with Linux. Broadcom NICs tend to flake
out around line rate and are also notorious for dropping packets in
hardware and not telling the kernel, so I try not to use those.


On Mon, Jan 9, 2017 at 3:31 AM, MrFastDie <nginx-forum at forum.nginx.org>
wrote:

> The OS of the server is debian 8, at my testing machine i'm using arch
> linux. Both NIC supports the speed of 1000Mb/s the server got round about
> 600 Mb/s up and 13Mb/s down.
>
> CJ Ess Wrote:
> -------------------------------------------------------
> > Which OS? What NIC? You also have to consider the traffic source, is
> > it
> > known capable of of saturating the NIC on your server?
> >
> > On Fri, Jan 6, 2017 at 10:24 AM, MrFastDie
> > <nginx-forum at forum.nginx.org>
> > wrote:
> >
> > > Hello,
> > >
> > > the last days I played a little with the NGINX settings and the tcp
> > stack
> > > to
> > > test the best performance. I used direct connection between my
> > testing
> > > machine and my server using a cat5e cable.
> > > My nginx.conf can be found at pastebin: http://pastebin.com/rRAEwvNc
> > > My sysctl.conf also provides lot of changes:
> > http://pastebin.com/KmPjEnHN
> > > My /etc/security/limits.conf provides soft nofile at 1048576 and
> > hard
> > > nofile
> > > at the same value.
> > >
> > > I'm not using any kind of firewall, all logs are disabled, neither
> > the cpu
> > > nor the ram is at it's limit, the i/o load seems fine to me too, but
> > the
> > > internet speed caps round about 600 Mbit/s. This test was made with
> > 2000
> > > connections per time and a 1MB file.
> > >
> > > The server contains an Intel(R) Pentium(R) Dual  CPU  E2220  @
> > 2.40GHz and
> > > has 3.5G of Ram.
> > > My local computer contains an Intel(R) Core(TM) i7-4710HQ CPU @
> > 2.50GHz and
> > > has 16G of Ram.
> > >
> > > The limits and tcp changes were also made on my local computer.
> > >
> > > Is there someone who can help me to deal with this problem?
> > >
> > > Posted at Nginx Forum: https://forum.nginx.org/read.
> > > php?2,271846,271846#msg-271846
> > >
> > > _______________________________________________
> > > nginx mailing list
> > > nginx at nginx.org
> > > http://mailman.nginx.org/mailman/listinfo/nginx
> > >
> > _______________________________________________
> > nginx mailing list
> > nginx at nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
>
> Posted at Nginx Forum: https://forum.nginx.org/read.
> php?2,271846,271869#msg-271869
>
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20170109/ced8e1c3/attachment-0001.html>