Apparently the frameworks desktop's 5g bit network isn't fast enough to scale well with LLM inference workloads, even for a modest GPU. Anyone know what kind of network is required to scale well for a single modest GPU?
In the case of llama.cpp's RPC mode, the network isn't the limiting factor for inference, but for distributing layers to nodes.
I was monitoring the network while running various models, and for all models, the first step was to copy over layers (a few gigabytes to 100 or so GB for the huge models), and that would max out the 5 Gbps connection.
But then while warming up and processing, there were only 5-10 Mbps of traffic, so you could do it over a string and tin cans, almost.
But that's a limitation of the current RPC architecture, it can't really parallelize processing, so as I noted in the post and in my video, it kinda uses resources round-robin style, and you can only get worse performance across the entire cluster than on a single node for any model you can fit on the single node.
Good news: USB4 mandates a direct host-to-host connectivity! Something it brought in from Thunderbolt. Hypothetically that should be 40Gbit connections, readily available.
I do hope that CXL 3.1 with its host to host capability makes gluess scale out easier. It's hyped as being for accelerators and attached memory, but having a much lower overhead RDMA capable fabric in every PCIe+CXL port is very very alluring. Can't come soon enough! Servers at first and maybe I'm hopelessly naive here but I do sort of expect it to show up on consumer too.
No network interconnect is going to scale well until you get into the expensive enterprise realm where infiniband and other direct connect copper/fiber reigns. The issue is less raw bandwidth but latency. Network is inherently 100x+ slower than memory access so when you start sharing a memory intensive workload like an LLM across a normal network it's going to crater your performance unless the work can be somewhat chunked to keep communication between nodes on the network to a minimum.
Really? Seems like scaling is pretty tolerant of latency, but very bandwidth intensive. Thus the move from IB to various flavors of ethernet (for AMD's GPUs, Tenstorrent, various others). Not to mention broadcom pushing various 50 and 100tbit ethernet switching chips for AI.
Even 25gbit these days is pretty affordable for home, if it scaled 5x better than 5gbit that might be enough to make larger models MUCH more practical.
It heavily depends on the workload, if one node needs to interact commonly with the memory on another node, like calculating the output of the weights stored on node the other node for the LLM, it's going to be dog slow because it has to wait 100x as long as it does for local. If you can batch the work into chunks that mostly get processed on one node then get passed to another then it can be parallelized easily.
eg if the individual layers of your model can fit on one node and the output can be pipelined so work can continue cascading through the various nodes it'd do well. But because the current word changes the next word a lot on LLMs you can't pipeline it. But you can see it in this [0] image from the attached blog post when he was testing llama.cpp, each node processes a batch of work and passes it off to the next node then goes idle.
