31 October 2020

TrueNas Scale Alpha: Infiniband support

With the launch of FreeNAS Scale's Alpha version I started digging into some of the things I had hoped a Linux launch would give us that the FreeBSD based version doesn't, and which have lead to me using Proxmox for my file sharing needs.

One of these is Infiniband support, specifically RDMA-over-Infiniband support. The former lacks the tooling in FreeNAS Core, the latter is not compiled in, as far as I'm aware.

I've been working on an article on Infiniband, and related technologies (CIFS over rDMA, and iSER, aka iSCSI over RDMA) for a while, so that will probably be finished eventually, so I won't got into detail on configuration here.

After installing TrueNAS Scale in a VM this is what I got:

truenas# lsmod|grep mlx
mlx4_ib               221184  0
ib_uverbs             163840  1 mlx4_ib
ib_core               397312  2 mlx4_ib,ib_uverbs
mlx4_core             376832  1 mlx4_ib

Well, that looks promising, let's see if the Infiniband tooling is available in the repositories:

apt-get update
apt-get install rdma-core infiniband-diags

That worked fine, so let's see what we have now:

truenas# modprobe ib_ipoib
truenas# lsmod|grep mlx
mlx4_ib               221184  0
ib_uverbs             163840  2 mlx4_ib,rdma_ucm
ib_core               397312  10 rdma_cm,ib_ipoib,rpcrdma,mlx4_ib,iw_cm,ib_iser,ib_umad,rdma_ucm,ib_uverbs,ib_cm
mlx4_core             376832  1 mlx4_ib

RDMA modules loaded, promising.

Let's see if everything is actually working, though at this point I'd have been surprised if they weren't.

truenas# iblinkinfo
CA: superbia HCA-1:
      0x0002c903004c1ba3      1    1[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       2    3[  ] "Infiniscale-IV Mellanox Technologies" ( )
CA: avaritia mlx4_0:
      0x0002c903002826db      4    1[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       2    2[  ] "Infiniscale-IV Mellanox Technologies" ( )
Switch: 0x0002c902004c8578 Infiniscale-IV Mellanox Technologies:
           2    1[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           2    2[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       4    1[  ] "avaritia mlx4_0" ( )
           2    3[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       1    1[  ] "superbia HCA-1" ( )
           2    4[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       5    1[  ] "truenas mlx4_0" ( )
           2    5[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           2    6[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           2    7[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           2    8[  ] ==(                Down/ Polling)==>             [  ] "" ( )
CA: truenas mlx4_0:
      0xe41d2d0300dc9771      5    1[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       2    4[  ] "Infiniscale-IV Mellanox Technologies" ( )
truenas# ibping -c 3 -G 0x0002c903002826db
Pong from avaritia.(none) (Lid 4): time 0.015 ms
Pong from avaritia.(none) (Lid 4): time 0.027 ms
Pong from avaritia.(none) (Lid 4): time 0.021 ms

--- avaritia.(none) (Lid 4) ibping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 3000 ms
rtt min/avg/max = 0.015/0.021/0.027 ms

And for kicks, and a little sneak peak about why Infiniband might be worth a bit of the pain to set it up: a gigabit Ethernet ping between the same two machines:

truenas# ping -c 3 192.168.0.104
PING 192.168.0.104 (192.168.0.104) 56(84) bytes of data.
64 bytes from 192.168.0.104: icmp_seq=1 ttl=64 time=0.233 ms
64 bytes from 192.168.0.104: icmp_seq=2 ttl=64 time=0.150 ms
64 bytes from 192.168.0.104: icmp_seq=3 ttl=64 time=0.137 ms

--- 192.168.0.104 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2044ms
rtt min/avg/max/mdev = 0.137/0.173/0.233/0.042 ms

Assuming iXsystems won't actively remove these features, and they will use the default Debian Samba/NFS packages (this will require more testing) this might finally mean TrueNAS with Infiniband/RDMA support for our homelabs!

Tags: Infiniband Networking GNU/Linux TrueNAS