Episode 6347 custom

Splitting LLM Layers Across AMD and Nvidia GPUs

A two-node llama.cpp setup running a 70B model jumps from three tokens per second on gigabit Ethernet to six on 2.5-gigabit.

Oct 29 21:27 Transcript

With

0:00

Episode Details

Episode ID: MWP-6549
Published: Oct 29
Duration: 21:27
Audio: Direct link
Pipeline: V5.25
TTS Engine: chatterbox-regular
Script Writing Agent: GPT-6.1 Sol
Topics: local-ai ai-inference gpus containers-and-virtualization

AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.

The prompt

The prompt that started this episode, as sent in (production notes for the hosts removed).

Say I have one computer with an AMD card and 10 GB of VRAM, and another computer with an Nvidia card and another 10 GB. Neither can fit the model I want, but together they could. Is there any framework that acts as a routing or pooling layer for inference, splitting one model's load across different hosts and different GPU vendors, so mismatched resources behave like one bigger GPU? If such a thing exists, how does it actually work, what are its limitations, does the model have to be loaded identically on both sides or can each machine hold a different slice, and is it ever worth doing in practice versus just buying one bigger card?

Clustering separate AMD and Nvidia machines makes it possible to run large models that exceed the VRAM of a single GPU. Using tools like llama.cpp with its ggml RPC backend, contiguous blocks of transformer layers can be assigned across disparate devices over a local network. Instead of constantly transmitting massive weight arrays, only intermediate activation tensors travel across the link between hosts. This setup bridges distinct vendor driver stacks by coordinating at the layer boundary rather than sharing unified GPU kernels. However, pooling memory solves capacity rather than execution speed. Performance heavily depends on network throughput, allocation heuristics like those in HiveLink or Parallax, and physical interconnect bottlenecks, which can sometimes slow interactive generation down to a crawl.

Sources

What the research for this episode read before the script was written. Primary sources first.

Context

Episode #5963 "Running One Model Across Mismatched GPUs" already covered the single-box case: tensor vs pipeline parallelism vs offloading, vLLM and llama.cpp support, mixed cards in one Linux machine. Don't re-explain those basics at length; this episode is about the cross-host, cross-vendor case: pooling memory over the network across AMD and Nvidia machines. Also worth distinguishing splitting one model across machines from simply load-balancing requests across separate full copies of a model.

Mentions

DRIFT Connects Mac MPS with CUDA over TCP
EXO Apple silicon cluster inference with Thunderbolt RDMA
frankenpool Mixed CUDA and Metal layer-split experiment
ggml Tensor library underlying llama.cpp
GLM 4.7 358B parameter model run across two machines
GLM-4.7 AI model from Z.AI for agentic code generation
HiveLink Wraps llama.cpp for mixed-node layer splitting
llama.cpp CPU inference engine with quantization support
Parallax Coordinates distributed layer ranges across nodes
Petals Public swarm for distributed layer serving
Skulk Apple, AMD, Nvidia nodes behind one endpoint

Downloads

Episode Audio

Download the full episode as an MP3 file

Download MP3

Transcript (TXT)

Plain text transcript file

Episode Book (PDF)

The episode's record — date, duration, models, sources — with the full transcript

#6347: Splitting LLM Layers Across AMD and Nvidia GPUs

The expensive part of a bigger graphics card is especially irritating when you already own enough memory, just in the wrong computers. Daniel wants to know whether an AMD machine with ten gigabytes of VRAM and an Nvidia machine with another ten can share one model that fits in neither. Can a routing layer divide the work, across both the network and the vendor divide, well enough to avoid buying another card?

Yes. The direct answer is llama.cpp with its RPC backend. AMD even publishes a playbook for clustering machines this way. You can give each machine a different portion of the model, rather than loading a complete copy on both.

Twenty gigabytes, separated by an argument.

Separated by Ethernet, preferably. Whether it is worth doing depends on the model, the link, and what you expect after it loads. Getting the weights into memory is the beginning of the engineering problem.

My expectation is that it answers before I forget why I asked. Also, those two ten gigabyte cards cannot spend their entire combined capacity on weights, can they? There is still the working memory.

There is. Context, temporary buffers, and runtime overhead all need room. Adding the labels on the boxes gives you a capacity ceiling, not a guaranteed model budget.

Two arrangements get mixed together here. Within one computer, Vulkan can let llama.cpp use Nvidia and AMD devices through a common backend. Across separate computers, RPC makes a remote device available over TCP. Daniel wants the second arrangement. Putting both cards in one chassis would change the transport problem, although it would not magically create direct shared memory.

You are already considering moving the cards. I can hear you measuring the case.

I was considering the power supply.

That is how your installations begin.

Fair. This one can stay on opposite desks. The RPC backend belongs to ggml, the tensor library underneath llama.cpp. It does no GPU computation itself. It forwards tensor operations to an RPC server on the other machine, where a local backend executes them. To llama-server, that remote backend looks like another device.

The coordinator sees a device. The user sees another computer that must stay awake, connected, and correctly installed. That distinction has ruined quite a few promises of simplicity.

Particularly when the remote computer is also somebody's everyday desktop. You need a compatible build and a working acceleration backend on each host. RPC bridges the machines. It does not install a missing driver or make an unsupported GPU supported.

Does AMD's own example actually depend on this, or is RPC an optional flourish beside a model that already fits?

It depends on the combined memory. Their playbook runs GLM four point seven, a three hundred fifty-eight billion parameter model, across two Ryzen AI Halo machines. The chosen quantization takes roughly two hundred five gigabytes on disk. They specify ten gigabit Ethernet, which tells you something about the intended environment.

Nobody in that diagram is trusting the kitchen WiFi.

The important expectation is capacity rather than acceleration. This lets you fit a larger model. It generally does not make the two machines faster than a suitable single device. Corn, would you accept a slower answer if it meant using the model you actually wanted?

For an overnight job, easily. For a conversation, there is a point where I start answering myself. That is an exceptionally low benchmark for the computer to miss.

Picture the model as its actual sequence of transformer layers. One host holds an early contiguous block. The other holds a later block. The first host processes its portion, then sends the resulting hidden-state activations across the network. The next host continues from there. The trained weights stay where their computation happens.

That last distinction matters. We are not sending ten gigabytes of weights back and forth for every word. We pay the loading cost, keep the slices resident, and then exchange the intermediate state.

Right... no, more precisely, there can be additional coordination and transfers depending on the graph and implementation. But the useful arrangement is to keep those contiguous blocks local, so the large weight arrays are not repeatedly travelling across the link.

No complete model on either GPU, then. Same model, different responsibilities.

Different resident layer ranges. You might still have a full model file on a host's disk because of how a particular loader works. Disk storage and GPU residency are separate questions. The inference arrangement does not require two complete copies in VRAM.

I was about to object to downloading half a model by cutting the file in half.

Please object. A model file has structure. The software assigns tensors and layers; you do not divide the download at a convenient byte.

HiveLink makes the arrangement particularly explicit. Its example sends work from a Windows machine with an RTX thirty eighty Ti to a Mac mini with an M four. CUDA handles the early layers, Metal the later ones. Activations cross the LAN between the blocks. Both sides use llama.cpp and GGUF models.

The AMD driver never has to understand a CUDA kernel. It only needs its own supported operation to consume the expected tensor. I suppose that is why the vendor boundary is tolerable here, while splitting one matrix multiplication between vendors would be much more awkward.

You have picked the important boundary. Inside a layer, tightly coordinated parallel computation can require frequent exchanges and compatible collective communication. Between blocks of layers, you have a clearer handoff. The machines agree on tensor contents and the model graph, rather than sharing a driver stack.

They agree on the work, not the manufacturer.

Yes, though backend support still matters. An operation or quantization supported on one device may not perform equally well on another. Mixed hardware expands what can participate. It also expands what you have to check.

Equal memory does not imply equal work either. Ten gigabytes on an older card and ten on a newer one could leave the newer card waiting for most of the answer.

HiveLink weights its allocation using the square root of VRAM multiplied by the logarithm of compute capability in teraflops. That is an allocation heuristic, not a guarantee of balance. Parallax also assigns layer ranges using memory, bandwidth, and compute. A faster node can take more layers, provided they fit.

Provided is doing honest work there. If both cards are almost full, there may be nowhere to move another layer, however badly we want to balance them.

Memory constrains the assignment before speed gets a vote. What would you optimize, Corn? Equal completion times, or the fewest handoffs?

Fewest handoffs first. Keep each machine's layers together, then adjust the boundary. Alternating layers between computers sounds like paying postage on every page.

That is a good instinct for this setting. For a single decoding stream, the next token depends on the previous token's result. The first machine doing its block and the second doing its block does not mean both are continuously busy. Multiple requests can provide more opportunities to overlap work, but that brings scheduling and memory demands of its own.

Hilbert's Anteater Trivia. Including its tail, a giant anteater can measure about two metres. I have never measured mine. It is not that kind of show.

I support the absence of measurement.

Keeping the handoffs together also explains the single-box contrast. One documented Vulkan setup combines a Tesla V one hundred with two Radeon AI Pro R ninety-seven hundred cards in a container. Vulkan exposes all of them, but transfers between those cards still bounce through host memory.

GPU to PCIe, system RAM, back across PCIe, then the other GPU. The chassis is smaller than a network, but proximity has not made them friends.

That particular cross-vendor setup has no direct peer-to-peer path. Across separate machines you add the network transport as well. Importantly, separate hosts already impose that boundary even if both GPUs have the same logo. The vendor mismatch is not what invents Ethernet latency.

There is also a pleasingly dreadful container detail in that example. The default Nvidia capabilities expose CUDA libraries without exposing the Vulkan pieces it needs. The Nvidia card can disappear from the Vulkan device list while sitting there perfectly capable of doing the work.

An excellent way to spend an evening diagnosing the wrong vendor. You have to expose the appropriate graphics capabilities, not merely establish that CUDA works.

One working driver, two incompatible definitions of working.

For the software landscape, HiveLink wraps llama.cpp for mixed nodes. Petals takes layer serving into a public swarm, where people host portions of supported models. That is a different trust and availability arrangement from Daniel's own two machines, even though the broad layer-sharing idea is related.

Load balancing full copies would not solve his problem at all. Sending alternate requests to each ten gigabyte card still leaves every request facing a model that does not fit.

Correct. Parallax coordinates distributed layer ranges. DRIFT connects a Mac using MPS with a CUDA machine over TCP, avoiding the usual distributed PyTorch communication stack. Skulk advertises Apple, AMD, and Nvidia nodes behind one compatible endpoint. Those are projects to examine, not interchangeable guarantees of support.

EXO is the name people are likely to offer before hearing the AMD part.

Its current headline path is MLX on Apple silicon with fast Thunderbolt communication. I would not choose it as the answer to this particular AMD and Nvidia pairing.

The performance trap is that a successful launch looks like success. You get the model name, both memory meters fill up, and only after pasting your actual document do you discover what you have built.

Especially because link speed alone cannot predict the result. Activations, synchronization, host copies, compute balance, and the RPC implementation all contribute. A faster cable removes one constraint. It does not remove every wait.

I would still want wired Ethernet before touching the layer split. WiFi makes the experiment depend on whatever else the building is doing.

Wired is the sensible baseline. One published two-node table for a quantized seventy billion parameter Llama reports about three tokens per second on gigabit Ethernet. Moving to two point five gigabit gets it to about six. That is a substantial difference in how reading the answer feels.

Half the waiting, without changing either GPU.

In the same table, ten gigabit reaches roughly seven and a half tokens per second, with Thunderbolt only slightly higher. The improvement flattens. Those results describe that setup, not a universal conversion between Ethernet speed and token speed.

You cannot buy ten times the Ethernet and invoice it for ten times the answer.

No. A mixed CUDA and Metal experiment called frankenpool is an even more sobering example. It runs a quantized seventy billion parameter DeepSeek distillation across an RTX forty seventy and an M one MacBook Air. Generation is about one point four tokens per second.

That gives punctuation time to reconsider.

Prompt processing was about three point two tokens per second. It used a forty dollar Thunderbolt cable. Cheap connectivity made the experiment possible, but neither the price nor the connector implies a fast conversational experience.

That model's weights are around forty gibibytes, though. It is not a demonstration that Daniel's twenty combined gigabytes can fit the same thing. Nor does its speed tell us exactly what his smaller model would do.

Those are both important corrections. We do not have an exact benchmark for his unspecified pair of ten gigabyte cards. The card models, chosen model, quantization, context, and backend all change the outcome. Single-digit generation speed is a possibility to budget for, not a measured prediction for his machines.

Hm.

Would that frankenpool speed be acceptable for your overnight use, or would the prompt processing sink it?

The prompt processing could absolutely sink it. If I feed it a long report, waiting for the first answer might cost more than generating the answer. An unattended job can be slow, but it still has to finish inside the time I actually have.

That is the prefill and decode distinction in practical terms. Prefill processes the incoming prompt, often with large batches of computation. Decode generates subsequent tokens step by step. A split that feels good after the first token can still make the initial wait miserable.

Your demonstration prompts are always suspiciously short, Herman.

They are controlled inputs.

Two words and a standing ovation.

I will use your long documents next time. One gigabit-linked Nvidia pair illustrates the compromise. Tuned for generation, it processed roughly three hundred fifty prompt tokens per second and generated thirty-six tokens per second. That is a responsive output stream once it starts.

Tuned toward prefill instead, prompt processing rose to about five hundred sixty tokens per second, while generation fell to nineteen. Same pair, different balance. Well... same hardware and reported workload conditions. The configuration is the thing being changed.

For a short exchange, I would keep the faster generation. For repeated long-document summaries, I might take the faster prefill, especially if the answers are short. Neither setting deserves to be labelled simply faster.

Context also consumes memory for the KV cache. A model that just squeezes into the combined cards at a short context may stop fitting at the length Daniel actually wants. Weight quantization and cache choices can help, but they change quality, support, or performance considerations.

Twenty gigabytes has acquired several tenants.

There is an implementation problem beyond those physical costs too. An open llama.cpp RPC issue reports remote GPU performance degradation, with a loss of up to fifty percent on an RTX fifty seventy Ti. The report says the penalty becomes more conspicuous on more capable GPUs.

Faster cards arrive earlier at the waiting.

Some of that cost may be reducible in software. We should not promote every observed slowdown into a law of distributed inference. Equally, fixing an RPC inefficiency does not eliminate transport, copies, or the sequential dependency of generating a single stream.

I would separate the purchase decisions. Already own both machines, have a wired link, need one model occasionally? Experiment. Buying a second underpowered machine specifically to avoid buying enough VRAM sounds like financing the inconvenience.

Power supplies, available slots, and physical space can make existing separate hosts attractive. People use RPC because they cannot sensibly add another card to the first machine. That is a real benefit even without faster generation.

There is a useful test before money enters it. Can a smaller or more aggressively quantized model do the actual job on one card? If yes, compare answers and elapsed time. Merely fitting the preferred model name is not yet an improvement.

I would compare a CPU-offloaded configuration too. Depending on system RAM and the machines involved, keeping part of the work in one host may be competitive with crossing the network. Not necessarily better... it is another baseline worth measuring.

Before you install four coordinators and a dashboard.

One coordinator. I have matured.

The fast homogeneous clusters show a different set of possibilities. EXO claims speedups with tensor parallelism over Thunderbolt five RDMA on Apple machines. Consistent hardware and a low-latency link let it pursue something quite different from two mismatched cards communicating over ordinary Ethernet.

Herman, can I keep the chat split and the long-document split as separate launchers? I have made both already. You were quite careful about saying which part got faster. My supplier was less careful. I paid him sixty-eight shekels for a network adapter before finding that the long prompt was what I could not stand.

Separate launchers make sense. They can preserve different allocations and settings without asking you to reconstruct them each time.

I have two office desktops in the cupboard. CPU only. The heavy one is slower. I have spent weeks moving the layer boundary by hand. Chat was quite tolerable, then I pasted a long document and went to the shop before it answered. Nissim has sold me cables for years. He took the adapter back. Kept twelve shekels for opening the packet. This week I am putting a little screen beside the cupboard so I can change launchers there. I have bought the bracket. The buttons will be large, since the little touch controls are no use to an anteater.

You are adding a screen to avoid walking to the computer that sent the request.

The cupboard is where I notice the noise. The launchers are called chat and homework. Those names stay. I write every speed in a notebook, although I have never opened it to choose a setting. The dashboard tells me what it thinks I want. It is rather fascinating that moving a few layers can change the wait before the first word so much. The screen cost more than the adapter. I am fitting it on Saturday.

Leave enough space for the cupboard door to open. You will still need access to both machines.

The bracket projects six centimetres. Nissim measured it against a bread knife.

The next useful improvement would be making those two workload settings less manual. Even if the RPC slowdown gets fixed, a long document and a short conversation still ask different things of the same machines. For Daniel, I would try the existing hardware before spending, but buy one adequately sized card if this becomes daily work.

Thanks to Hilbert Flumingtop for producing the show.

For more along these lines, there's episode two, Local STT For AMD GPU Owners; episode seventeen, Cloud Render Superpowers; and episode three, Safetensors or something else. This has been My Weird Prompts, the human-AI collaboration podcast. No question too obscure, no rabbit hole too deep.

If you enjoyed this one, leave us a review. It helps other curious people find the show.

See you tomorrow.

Until then.

This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error