7

I have a server running ubuntu, I have 10G card configured with mtu 9000 and a NFS server has also a 10G card and is directly connected to the server with a dedicated link (no switch in between). The ubuntu server has following settings

enp66s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        inet 192.168.1.2  netmask 255.255.255.252  broadcast 192.168.1.3
        inet6 fe80::d65d:64ff:fe09:b66  prefixlen 64  scopeid 0x20<link>
        ether d4:5d:64:09:0b:66  txqueuelen 1000  (Ethernet)
        RX packets 3065123981  bytes 3717515096214 (3.7 TB)
        RX errors 1963997  dropped 0  overruns 1887746  frame 76251
        TX packets 2720860289  bytes 4802586105745 (4.8 TB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 99  memory 0x20080800000-20080ffffff

And the nfs server has following configuration,

bnxt2: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000     options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,
LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether 84:16:0c:aa:6b:32
        inet 192.168.1.1 netmask 0xfffffffc broadcast 192.168.1.3
        media: Ethernet autoselect (10GBase-CR1 <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

I do not get or I am not sure if I get the 10G connectivity, when I copy some data using dd command I get following from bmon. Please take a look and explain if I am making use of 10G in the optimized way or not. This image shows the nfs client bmon

This image shows the nfs server bmon

This image shows the nfs client running dd command

AAB
  • 115
  • 8

1 Answers1

13

Note the capital B in 1185 / 1167 MiB/s in your screenshots - 1192 MiB/s or 1250 MB/s is the theoretical maximum of a 10 Gbit/s Ethernet connection, disregarding any L2/L3/L4 overhead.

You're very close, due to the use of jumbo frames. Using NFS over TCP over IPv4 over 10GE, you can calculate a maximum achievable throughput of 1186.6 MB/s with the standard 1500 byte MTU, and 1239.2 MB/s with 9000 byte MTU.

Using binary MiB/s isn't very useful for throughput calculation, but if need be multiply the above by 0.9547.

Zac67
  • 13,684