Daniel sent us this one — he points out something genuinely absurd when you stop and think about it. We talked recently about organizations moving data by truck because even a forty gigabit per second connection isn't fast enough. And he asks: if forty gig is considered a bottleneck, what speeds do government agencies and scientific organizations actually need? What specialized hardware makes those speeds possible? And what's the absolute physical maximum for network throughput right now?
The first thing to understand is that forty gig is basically a dial-up modem in the world we're about to describe. I mean that almost literally. When NOAA is running hurricane models, they're shoving around datasets that make forty gig look like someone's trying to drain a lake through a garden hose.
The garden hose of national infrastructure.
And the gap is getting wider, not narrower. Consumer internet has basically plateaued around what people actually need — ten gig is overkill for Netflix. But on the institutional side, AI training clusters and climate models are generating datasets that double in size every couple of years. So you've got this divergence where home users are fine at one gig and scientists are choking at four hundred gig.
Let's put some actual numbers on what "fast" means when your dataset is measured in petabytes.
The speed ladder we're talking about starts at forty gig — which is where that climate organization was stuck — then jumps to one hundred gig, which is table stakes for any serious research network. Then four hundred gig, which is where the cutting edge currently operates. Then eight hundred gig, which is rolling out now. Then one point six terabit, which is the bleeding edge. And three point two terabit is on the horizon.
Just so listeners can feel the difference: transferring one petabyte at forty gig takes about fifty-five hours. At four hundred gig, it's five and a half hours. At one point six terabit, it's about an hour and twenty minutes.
That's assuming perfect conditions, no protocol overhead, no congestion. In reality, you're probably adding twenty percent to those numbers. But the point stands: going from forty gig to four hundred gig takes you from "this is a multi-day project" to "I can do this before lunch.
Which is still wild when you think about it. A petabyte before lunch. I remember when a megabyte before lunch was aspirational.
You're dating yourself.
I'm a sloth. Dating myself is kind of the point.
So let's talk about who actually needs these speeds and why. The Department of Energy runs something called ESnet — the Energy Sciences Network. It connects all the national labs: Oak Ridge, Argonne, Lawrence Berkeley, Los Alamos, all of them. ESnet6, which is the current generation, operates at four hundred gigabits per second across the backbone, and they've already got plans to upgrade to eight hundred gig by twenty twenty-seven.
This isn't theoretical. This is live, operational infrastructure moving real data right now.
The ESnet6 upgrade was completed in late twenty twenty-two, and it carries something like one point six exabytes of traffic per year. That's across something like sixteen thousand miles of dedicated fiber. This is not shared infrastructure — the DOE literally owns or leases dark fiber strands that are used exclusively for science data.
There's something deeply satisfying about the phrase "dark fiber strands used exclusively for science data." It sounds like the opening of a novel I'd actually want to read.
The other big player is CERN. The Large Hadron Collider generates something on the order of ninety petabytes of data per year during its runs. That data has to get from the detectors in Switzerland to computing centers all over the world for analysis. CERN built a dedicated network called the LHCOPN — the LHC Optical Private Network — and the backbone links are one hundred gigabits per second each, with multiple parallel links.
They're moving over a hundred petabytes a year across that network.
And here's the thing that most people don't appreciate: the LHC doesn't just produce data and ship it. It produces data continuously during runs, and the Tier Zero computing center at CERN does an initial processing pass, then ships the results to eleven Tier One centers around the world. Those centers do further processing and ship to over a hundred and fifty Tier Two sites. So you've got this multi-tiered distribution tree where every link has to handle enormous sustained throughput.
It's not just a one-time migration. It's a constant churn of data flowing through this hierarchy.
And then you've got NOAA, which runs operational weather models — not research, operational, meaning if these models don't run, your hurricane forecasts are degraded. NOAA's backbone connecting their supercomputing centers runs at four hundred gigabits per second. They upgraded to that in twenty twenty-three specifically because their previous one hundred gig links were becoming a bottleneck during ensemble model runs.
Ensemble runs being where they run the same model dozens of times with slightly different initial conditions to produce a probability distribution.
And each ensemble member produces a full dataset. So if you're running fifty ensemble members and each one produces ten terabytes of output, you've just generated five hundred terabytes that need to move somewhere for post-processing and visualization. At one hundred gig, that's about eleven hours. At four hundred gig, it's under three.
Which matters when you're trying to get a hurricane forecast out before the hurricane actually arrives.
There's a concept called "time to insight" that drives all of this. It's not just about raw throughput — it's about how long it takes from when the data is generated to when a scientist can actually look at it and make a decision. For weather forecasting, every hour matters. For the LHC, every day of delay in data processing means physicists are waiting to test their theories. For the DOE labs, if you're running a multi-million-dollar supercomputer and it's sitting idle waiting for data, you're burning money.
The idle supercomputer is the white elephant of scientific computing.
It really is. Frontier at Oak Ridge cost something like six hundred million dollars. If it's sitting there waiting for data because your network is too slow, that's a very expensive paperweight.
We've established that forty gig is slow, four hundred gig is the new normal for serious work, and the real action is happening at eight hundred gig and above. Now let's talk about the hardware, because this is where it gets fascinating. Every single component in the chain has to be purpose-built.
This is the part I love. Because when people think about network speed, they think about the cable. "Oh, it's fiber, it's fast." But the cable is almost never the bottleneck. The bottlenecks are the transceivers, the switch silicon, the network interface cards, and — crucially — the PCIe bus inside the server.
The PCIe bus. The thing most people never think about unless they're building a gaming PC, and even then they're mostly worried about GPU lanes.
So let's walk through the chain. At the server end, you've got a network interface card — a NIC. For consumer stuff, you're looking at a ten gig NIC that costs maybe fifty dollars and uses a single PCIe lane. For four hundred gig and above, you're looking at something like the NVIDIA Mellanox ConnectX-7 or ConnectX-8. The ConnectX-7 supports four hundred gigabit Ethernet and InfiniBand. The ConnectX-8, which started sampling in twenty twenty-four, supports eight hundred gig.
These are not fifty dollars.
These are thousands of dollars per card. And they're not just dumb interfaces. They've got onboard processors that handle RDMA — Remote Direct Memory Access — which allows data to bypass the CPU entirely and go straight from the network into application memory. They've also got something called GPUDirect, which lets the NIC write directly into GPU memory.
The data goes from the wire straight into the GPU without the CPU ever touching it.
And that matters because at four hundred gigabits per second, if the CPU had to handle every packet, you'd need dozens of cores just dedicated to network processing. The CPU becomes the bottleneck, not the network.
Which is also why these cards consume twenty-five to thirty-five watts just for the network interface. That's more power than some entire low-end servers.
That heat has to go somewhere. Data centers that deploy eight hundred gig NICs at scale have to completely rethink their cooling. But the NIC is just the first link in the chain. The next bottleneck is PCIe. A single four hundred gig NIC requires eight lanes of PCIe 4.0, or four lanes of PCIe 5.An eight hundred gig NIC requires sixteen lanes of PCIe 5.0 — that's the entire lane budget for a typical single-socket server.
You literally can't put two eight hundred gig NICs in most servers because there aren't enough PCIe lanes.
And that's why PCIe 6.0 is such a big deal. It doubles the per-lane bandwidth to sixty-four gigatransfers per second, which means you can support a one point six terabit NIC on sixteen lanes. 0 hardware is just starting to ship now, and it's going to be essential for the next generation.
The NIC plugs into the server, and then the NIC plugs into... What's on the other end of that cable?
The cable goes to a switch, and the switch is where the real magic happens. The heart of every high-speed switch is the ASIC — a custom silicon chip that does the actual packet switching. The current king is the Broadcom Tomahawk 5, which has a switching capacity of fifty-one point two terabits per second and can support sixty-four ports of eight hundred gig each.
Fifty-one terabits. In a single chip.
In a single chip the size of your palm. And it's not just Broadcom — Cisco has their Silicon One G200, also fifty-one point two terabits. Marvell has the Teralynx 10, same capacity. These chips are basically small supercomputers dedicated to one task: looking at a packet header and deciding which port to send it out, billions of times per second.
The chip the size of your palm that can route the entire internet traffic of a small country.
They do it with astonishingly low latency. We're talking hundreds of nanoseconds from ingress to egress. The Tomahawk 5 can process something like twenty-five billion packets per second.
At that scale, the speed of light inside the chip actually starts to matter. The distance the electrons have to travel becomes a design constraint.
Chip layout becomes a physics problem. You have to account for signal propagation delay across the die. But let's talk about the cables themselves, because this is where a lot of misconceptions live. At four hundred gig and above, you're almost exclusively using single-mode fiber — specifically OS2 fiber, which has a nine-micron core. This is the same type of fiber that's been used for long-haul telecom for decades.
The fiber itself isn't exotic. The magic is in the transceivers at each end.
The transceiver is the thing that converts electrical signals to light and back. At four hundred gig, you're using form factors called QSFP-DD — Quad Small Form Factor Pluggable Double Density — or OSFP, which is Octal Small Form Factor Pluggable. These are little metal boxes about the size of a pack of gum that plug into the switch or the NIC.
Inside that pack of gum is basically a tiny laser and a tiny photodetector and some very sophisticated signal processing.
The current standard for one hundred gig per lane uses something called PAM4 signaling — four-level pulse amplitude modulation. Instead of just on and off, you've got four distinct signal levels, which lets you encode two bits per symbol instead of one. That's how you double the data rate without doubling the frequency.
You're not making the laser blink faster, you're making it blink with more nuance.
That's a wonderfully sloth way to put it. Yes, more nuanced blinking. And the IEEE standard for one point six terabit — 802.3ck — uses one hundred and twelve gigabit per second PAM4 lanes. You run eight of those lanes in parallel, and you get eight hundred gig. Run sixteen lanes, you get one point six terabit.
This was finalized when?
3ck standard was finalized in late twenty twenty-three, and the first products using it shipped in twenty twenty-four. One point six terabit transceivers in the OSFP form factor started appearing in early twenty twenty-five. Microsoft deployed eight hundred gig for their Azure AI supercomputers in twenty twenty-five as well.
We're not talking about lab experiments. This is shipping product.
And the next step is already being worked on — IEEE 802.3df, which will use two hundred and twenty-four gigabit per second PAM4 signaling to hit three point two terabits per port. That's probably going to be standardized by twenty twenty-six or twenty twenty-seven.
There's something almost absurd about the pace. By the time most enterprises have upgraded to one hundred gig, the cutting edge is already at three point two terabit.
That's actually a great thing for the homelab crowd and small businesses. Because as the national labs and hyperscalers upgrade from one hundred gig to four hundred gig and eight hundred gig, all that one hundred gig gear floods the secondary market. You can get a Mellanox ConnectX-4 one hundred gig NIC on eBay right now for under a hundred dollars.
The trickle-down of networking. Like how last decade's supercar becomes this decade's affordable sports car.
And it's not just NICs. One hundred gig switches that cost thirty thousand dollars new five years ago are showing up on the used market for under two thousand. That's still not cheap for a home user, but for a small business or a serious homelab, it's actually within reach.
The listener who's been following along thinking "this is fascinating but irrelevant to me" — there's actually a practical angle here. The gear that's obsolete for CERN is aspirational for your basement rack.
And twenty-five gig is becoming the new sweet spot for homelab. You can get twenty-five gig NICs for under fifty dollars, and switches are starting to appear in the sub-five-hundred-dollar range. That's ten times faster than the two-and-a-half gig that most consumer motherboards ship with, for roughly the same cost if you're willing to buy used enterprise gear.
It's the circle of life, but with more blinking lights.
Now let's talk about what happens when you push all this to the physical limit. Because there is a limit. Single-mode fiber is incredibly good — you can push hundreds of terabits per second through a single strand in theory — but you run into the Shannon limit eventually. That's the fundamental information-theoretic maximum for how much data you can push through a channel with a given bandwidth and signal-to-noise ratio.
The universe's way of saying "that's enough.
For a standard single-mode fiber with optical amplification, the practical limit is somewhere around one hundred terabits per second per fiber. Beyond that, you start hitting nonlinear effects in the glass — the light literally starts interacting with itself, causing distortion that no amount of signal processing can fully correct.
We're not going to see petabit-per-second single-fiber links. The physics just says no.
Not with current fiber technology. You can multiplex across multiple cores in a multi-core fiber, or use multiple fibers in parallel, which is what everyone does. But single-fiber, single-core, we're probably within a factor of ten or twenty of the theoretical maximum.
Which is kind of comforting in a way. There's a ceiling. We know where it is. We can plan around it.
That ceiling is still mind-bogglingly high. One hundred terabits per second is enough to transfer the entire printed collection of the Library of Congress in about two seconds. The practical limits today aren't the fiber, they're everything at either end — the transceivers, the switch silicon, the PCIe buses, the storage systems that have to actually write the data to disk.
That's the other bottleneck nobody talks about. You can receive data at eight hundred gigabits per second, but can your storage array actually write it that fast?
A single NVMe drive can write at maybe seven gigabytes per second — that's about fifty-six gigabits per second. So to saturate an eight hundred gig link, you need about fifteen NVMe drives writing in parallel. That's a serious storage array. And if you're writing to spinning rust, forget it — you'd need hundreds of drives.
The network is only as fast as the slowest component in the entire pipeline. NIC, PCIe, switch, transceiver, fiber, transceiver, switch, PCIe, NIC, storage. Any one of those becomes the bottleneck, and you're limited to whatever it can handle.
This is why organizations like the DOE and CERN don't just buy fast switches. They have to engineer the entire data path end to end. Every server, every storage node, every switch, every fiber span has to be provisioned and tested to handle the full line rate. It's systems engineering at a scale that most enterprises never have to think about.
Let's put some concrete numbers on what this looks like in practice. Say you're NOAA and you need to move five hundred terabytes of ensemble model output from your supercomputer in Reston, Virginia to your visualization center in Boulder, Colorado. That's about fifteen hundred miles. At four hundred gig, the transfer takes about two hours and forty-five minutes — but that's just the raw transmission time. Add in protocol overhead, congestion from other traffic, and the fact that TCP doesn't play nice with long fat pipes unless you tune it carefully, and you're probably looking at more like three and a half hours.
That's where WAN optimization and parallel streams come in. The speed of light in fiber is about two hundred thousand kilometers per second, which means the round-trip time from Reston to Boulder is about twenty-four milliseconds. At four hundred gig, the bandwidth-delay product — the amount of data in flight — is about one point two gigabytes. That's how much data you have to keep "in the pipe" to fully utilize the link.
If your TCP window isn't sized correctly, you're leaving capacity on the table.
This is why protocols like GridFTP and tools like Globus were developed specifically for science data transfer. They open multiple parallel TCP streams and tune the window sizes automatically. A single TCP stream at four hundred gig with twenty-four milliseconds of latency will struggle to fill the pipe. Eight parallel streams will saturate it easily.
It's not just about buying fast hardware. You need the software stack to match.
The expertise to configure it. Which is why the DOE runs a dedicated network engineering team at ESnet, and CERN has a whole group dedicated to data movement. This is not "plug it in and it works" territory.
The plug-and-play era ends somewhere around ten gig.
Honestly, ten gig is still pretty plug-and-play these days. The real pain starts at one hundred gig and gets exponentially worse from there. At four hundred gig, you're dealing with forward error correction algorithms, PAM4 signal integrity issues, and thermal management that requires active cooling on the transceivers.
Active cooling on something the size of a pack of gum.
Those QSFP-DD transceivers can draw up to fifteen watts each. A fully loaded sixty-four port switch with eight hundred gig optics is dissipating close to a kilowatt just in the transceivers. That's before you account for the switch silicon itself.
Which brings us back to the physical limits question. What is the absolute maximum throughput you can get on a single port right now, with hardware you can actually buy?
Right now, mid twenty twenty-six, the answer is one point six terabits per second. That's using OSFP transceivers with eight lanes of two hundred gig PAM4, or sixteen lanes of one hundred and twelve gig PAM4, depending on the implementation. There are shipping products from multiple vendors — NVIDIA, Intel, Broadcom, Marvell. The switches exist, the NICs exist, the transceivers exist.
For a single one point six terabit switch port with the transceiver, you're looking at somewhere north of ten thousand dollars. The switch itself — a chassis that can support, say, thirty-two of those ports — is a couple hundred thousand dollars. This is not consumer gear. This is not even enterprise gear. This is hyperscaler and national lab territory.
For the listener who's been thinking "maybe I'll put one point six terabit in my homelab" — maybe don't.
Maybe wait five years. But here's the thing that I find exciting: the trajectory is clear. One point six terabit is the new eight hundred gig, which was the new four hundred gig. By twenty twenty-eight, one point six will be the standard for backbone networks at the DOE and similar organizations. Three point two terabit will be the cutting edge. And all the four hundred gig and eight hundred gig gear will be cascading down to the enterprise market, and eventually to the used market where homelab enthusiasts can get their hands on it.
The velocity of obsolescence is a gift to the patient.
And there's one more thing I want to mention about the physical limits, because it connects back to something we said earlier. The ultimate bottleneck for single-fiber links isn't the Shannon limit, practically speaking — it's the fact that beyond a certain point, you can't make the transceivers work fast enough. The laser has to switch between four distinct amplitude levels billions of times per second. At two hundred and twenty-four gig per lane, the symbol period is about nine picoseconds. That's nine trillionths of a second. The electronics have to be unbelievably precise.
At some point, you're fighting the uncertainty principle.
You're not far off. The noise floor becomes a fundamental physics problem. And that's why the industry is moving toward things like co-packaged optics, where the optical engine is integrated directly onto the switch ASIC package, eliminating the electrical interface between the switch and the transceiver entirely.
Instead of a pluggable transceiver, the laser is literally built into the chip package.
And that eliminates several inches of copper trace on the circuit board, which at these speeds represents a significant signal integrity challenge. Co-packaged optics are going to be essential for three point two terabit and beyond. The first commercial products using CPO are expected in the next year or two.
It's amazing how much of ultra-high-speed networking comes down to "make the wires shorter.
That's honestly a pretty good summary of the entire field. Shorter wires, better signal processing, more parallel lanes. The fundamental insight hasn't changed in decades — the implementation just keeps getting more sophisticated.
To pull this all together for the listener who's been tracking with us: the government agencies and scientific organizations we're talking about operate at speeds that are effectively a different category of technology from what most people experience as "fast internet." Forty gig is a bottleneck. Four hundred gig is the baseline. Eight hundred gig is current. One point six terabit is the bleeding edge. And three point two terabit is coming.
The hardware that makes it possible — the NICs with RDMA and GPUDirect, the switch ASICs that switch fifty-one terabits per second, the single-mode fiber with PAM4 modulation, the PCIe 6.0 buses — is almost entirely invisible to consumers. It lives in data centers and research labs, and it's the reason your weather forecast is accurate and the LHC can discover new particles.
The invisible infrastructure of modern science. All those blinking lights in windowless rooms, pushing photons through glass at the edge of what physics allows.
The practical takeaway for anyone listening who's building a network: keep an eye on the secondary market. As the labs upgrade to four hundred gig and eight hundred gig, one hundred gig gear is becoming affordable. Twenty-five gig is the new sweet spot for homelab. You can build a network that would have been science fiction fifteen years ago for a few hundred dollars and some eBay patience.
Which is honestly the best kind of trickle-down. Not economic theory, just cheaper switches.
The only kind of trickle-down that actually works.
The open question I keep coming back to is whether consumer internet will ever catch up. Right now, the gap between "fast enough for Netflix" and "fast enough for science" is widening, not narrowing. Is there a future where home users have any reason to need four hundred gig?
I don't think so, honestly. The applications that drive consumer bandwidth demand — video streaming, video calls, game downloads — are plateauing. Eight K video requires about a hundred megabits per second. Even with multiple streams, a one gig connection is plenty for the foreseeable future. The institutional side is driven by completely different forces: AI training, scientific simulation, genomics. Those aren't consumer applications and probably never will be.
We're looking at a permanent fork in the networking road. Consumer internet stays in the single-digit gigabit range, institutional networking climbs toward the terabit range, and the two worlds just...
That's fine. The institutional gear eventually trickles down to enterprise, and enterprise gear trickles down to prosumer and homelab. The consumer market gets the benefits indirectly — better cloud services, more accurate weather forecasts, faster scientific discoveries — without ever needing to plug in a four hundred gig transceiver.
The network that serves you is fast even if the network in your house isn't.
And that's been the model for decades. It's just that the gap between "serves you" and "in your house" has gotten dramatically wider.
I think that's a good place to leave it. The snowmobile episode taught us that sometimes a truck full of hard drives is faster than the internet. This episode taught us that even when the internet is absurdly fast, it's still not fast enough for the people pushing the boundaries of what we know.
And now: Hilbert's daily fun fact.
Hilbert: In the nineteen twenties, the largest traditional indigo dyeing vat in the Simpson Desert region was a hand-dug clay pit measuring thirty-one feet in diameter, used by Aboriginal communities to process roughly nine hundred pounds of plant material per batch — producing enough dye to color approximately four thousand square feet of fabric in a single cycle.
...thirty-one feet of indigo in the desert.
I have so many questions, none of which I'm going to ask.
This has been My Weird Prompts. Our producer is Hilbert Flumingtop. You can find every episode, show notes, and transcripts at myweirdprompts dot com.
If you enjoyed this episode, leave us a review wherever you get your podcasts — it helps other people find the show. We'll be back next week with another prompt.
