5. Thunderbolt Network config
Tests
This part has been a bugger. Having multiple IP addresses messes with the MPI Hydra spawner. So let's say the mini1 has a DHCP address of 192.168.7.11, and mini2 is 192.168.7.22. Initially I tried having a directly connected subnet, say 10.1.1.0/30, mini1: 10.1.1.1, mini2: 10.1.1.2. Override the host in /etc/hosts does NOT propigate through to the slurm controller. I.e. when MPI is spawned, mini1 and mini2 are connected via 192.168.7.0 subnet, which is the slow ethernet link.
I've tried noodling with MPI hydra's spawner and haven't made any headwind.
Partially working setup
My next attempt is to create a bridge between the ethernet NIC and the thunderbolts. To do this: create bridge between eth0, thunderbolt0-3. Each node get's a single IP address via DHCP and the bridge will automatically route to neighbors.
Now when launching MPI jobs through slurm, they do in fact initialze directly. Pt2pt bandwidth tests yielded the expected performance (2G/s), but when I ran bi-directional benchmarks something didn't add up.
Take a look at this:
[noah@supermini ~]$ mpirun -np 2 -ppn 1 -hosts mini3,mini4 -bind-to=core osu_bibw # OSU MPI Bi-Directional Bandwidth Test v7.4 # Datatype: MPI_CHAR. # Size Bandwidth (MB/s) 1 0.27 2 0.83 4 1.67 8 3.48 16 6.53 32 12.81 64 26.37 128 52.62 256 103.70 512 146.80 1024 213.20 2048 402.98 4096 755.64 8192 1351.20 16384 1789.00 32768 1906.85 65536 1951.00 131072 1863.27 262144 1977.60 524288 2016.45 1048576 2005.80 2097152 2045.44 4194304 2047.36 [noah@supermini ~]$ mpirun -np 2 -ppn 1 -hosts mini1,mini5 -bind-to=core osu_bibw # OSU MPI Bi-Directional Bandwidth Test v7.4 # Datatype: MPI_CHAR. # Size Bandwidth (MB/s) 1 0.37 2 1.12 4 2.25 8 4.37 16 9.35 32 18.38 64 35.97 128 61.32 256 125.53 512 183.96 1024 246.68 2048 485.59 4096 911.34 8192 1641.18 16384 2670.24 32768 3181.92 65536 3385.47 131072 3116.69 262144 3257.41 524288 3345.75 1048576 3392.24 2097152 3397.04 4194304 3224.09
Turns out this is a side effect of the bridge creating incomplete routes with STP.
[root@mini1 ~]# bridge link 2: enp4s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state blocking priority 33 cost 150 4: thunderbolt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 master br0 state forwarding priority 32 cost 100 5: thunderbolt1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 master br0 state forwarding priority 32 cost 100 7: thunderbolt2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 master br0 state forwarding priority 32 cost 100 8: thunderbolt3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 master br0 state forwarding priority 32 cost 100 [noah@mini2 ~]$ bridge link 3: enp4s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state blocking priority 33 cost 150 4: thunderbolt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 master br0 state forwarding priority 32 cost 100 5: thunderbolt1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 master br0 state forwarding priority 32 cost 100 6: thunderbolt2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 master br0 state forwarding priority 32 cost 100 9: thunderbolt3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 master br0 state blocking priority 32 cost 100 [noah@mini3 ~]$ bridge link 9: thunderbolt2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 master br0 state forwarding priority 32 cost 100 10: thunderbolt3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 master br0 state forwarding priority 32 cost 100 11: thunderbolt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 master br0 state blocking priority 32 cost 100 12: thunderbolt1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 master br0 state blocking priority 32 cost 100 [noah@mini4 ~]$ bridge link 2: enp4s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state blocking priority 33 cost 150 5: thunderbolt1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 master br0 state forwarding priority 32 cost 100 9: thunderbolt3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 master br0 state blocking priority 32 cost 100 10: thunderbolt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 master br0 state blocking priority 32 cost 100 11: thunderbolt2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 master br0 state blocking priority 32 cost 100 [noah@mini5 ~]$ bridge link 3: enp4s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 33 cost 150 5: thunderbolt1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 master br0 state forwarding priority 32 cost 100 9: thunderbolt3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 master br0 state forwarding priority 32 cost 100 10: thunderbolt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 master br0 state forwarding priority 32 cost 100 11: thunderbolt2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 master br0 state forwarding priority 32 cost 100