6. Thunderbolt Network Round 2

Each port will have it's own interface and be directly connected to another node via /30 subnet. Set MTU 65520

Linux routing to funnel trafic to certain host.

systemctl enable systemd-networkd.service systemctl disable NetworkManager.service

systemctl reload systemd-networkd.service

sudo emacs /etc/systemd/network/20-wired.network
[Match]
Name=enp4s0f0

[Network]
DHCP=yes

[DHCPv4]
RouteMetric=100

udevadm monitor

SUBSYSTEM=="net", DEVPATH=="/devices/pci0000:00/0000:00:01.2/0000:7b:00.0/0000:7c:00.0/0000:7d:00.0/domain1/1-0/1-1/*", NAME="big4"

[Match]
OriginalName=thunderbolt*

[Link]
MTUBytes=65520
TransmitQueueLength=2424

[Match]
Name=mini4

[Address]
Address=10.4.5.2/30

#route add -host 192.168.7.181 dev mini5 #route add -host 192.168.7.160 dev mini4 route add -host mini5 dev mini5 route add -host mini4 dev mini4 route add -host mini3 dev mini3 route add -host mini2 dev mini2 route add -host mini1 dev mini1

su - root salloc -N5 srun route add -host mini5 dev mini5 srun route add -host mini4 dev mini4 srun route add -host mini3 dev mini3 srun route add -host mini2 dev mini2 srun route add -host mini1 dev mini1

Turns out to be pretty painless

$ ml mpich omb
$ salloc -N2 --nodelist=mini4,mini5
$ mpirun -np 2 -ppn 1 osu_bibw
# OSU MPI Bi-Directional Bandwidth Test v7.4
# Datatype: MPI_CHAR.
# Size      Bandwidth (MB/s)
1                       0.29
2                       1.13
4                       2.32
8                       4.63
16                      9.17
32                     18.09
64                     37.21
128                    71.50
256                   141.46
512                   191.84
1024                  255.95
2048                  493.31
4096                  904.16
8192                 1727.58
16384                2827.67
32768                3495.95
65536                3604.73
131072               3508.72
262144               3738.09
524288               3884.12
1048576              3939.22
2097152              3930.06
4194304              3912.45

$ salloc -N5
$ mpirun -np 5 -ppn 1 osu_alltoall
# OSU MPI All-to-All Personalized Exchange Latency Test v7.4
# Datatype: MPI_CHAR.
# Size       Avg Latency(us)
1                     151.96
2                     150.78
4                     150.58
8                     153.47
16                    155.04
32                    150.15
64                    152.39
128                   151.22
256                   151.66
512                   152.83
1024                  158.66
2048                  160.11
4096                  175.44
8192                  178.75
16384                 201.50
32768                 218.54
65536                1455.05
131072               4394.21
262144               4604.17
524288               4840.48
1048576              5758.82

pacman -Rs networkmanager

systemctl disable ntpd.service systemctl stop ntpd.service pacman -Rs ntp systemctl enable systemd-timesyncd systemctl start systemd-timesyncd

https://github.com/torvalds/linux/blob/09f6b0c8904bfaa1e0601bc102e1b6aa6de8c98f/drivers/thunderbolt/xdomain.c#L424C23-L424C33

"Currently we expect all requests to be directed to us. The protocol supports forwarding, though which we might add support later on."

The underlying protocol of thunderbolt is called XDomain. It's confirmed that it does in fact allow forwarding, however the code is not yet in the linux kernel. This would enable daisy chaining thundebrolt connections, freeing up a lot of open ports to expand the cluster with misc thunderbolt machines I have.

I.e. Mini1<->Mini2<->Mini3 In theory Mini3 could send request to Mini1 and the thunderbolt fabric would handle switching, not interrupting Mini2's CPU.