Daniel sent us a prompt that picks up right where we left off on proxies, and I'm going to read it because he's asking something I genuinely didn't know the answer to. Here's what he wrote.
"We talked about proxies for Claude Code in a previous episode — projects that intercept API requests to write to local resources and other use cases. Herman made a very good point that, rather than vendors seeing proxies as threats, they should realize that they're just an expression of user demand for more developed APIs. The long-term trend might be that proxy layers become totally normalized, much as load balancers are in cloud infrastructure. A related point I'd love to talk about is how APIs are actually managed at scale, because we talked about API gateways. Users who've poked around AWS dashboards have seen how load balancing works and load balancers that route traffic onto servers, but the usual assumption for that diagram is that traffic hits the API. This makes sense if you're a small business. But if we take, say, the OpenAI API or the Anthropic API, we're talking about an endpoint that's presumably being hit by millions or tens of millions of users almost continuously. At that scale, how and where the API is hosted and how load balancing happens at that layer presumably becomes a major point of architecture, much as load balancing onto the actual servers is too. Other than playing around with tools like Kong, however, I've never actually seen how a public-facing API is managed, and certainly not at that scale. What kind of tooling is deployed here to make this important part of web plumbing actually work?"
Okay, there's a lot in there. And Herman, you did make that point about proxies being demand signals — I remember you saying it and I remember thinking, yeah, that's exactly right, and now Daniel's using it as a launchpad for a much bigger question about what's actually happening on the other side of the API endpoint.
I did say that, and I stand by it. But Daniel's question is the one I wish more developers asked — because most people treat the API as this magic curtain. You send a request, you get a response, and everything in between is someone else's problem. At the scale of OpenAI or Anthropic, that "in between" is a distributed system in its own right.
So let's start with what most of us think an API gateway looks like, and then we'll blow that model up.
Right. The mental model most developers carry around is a single reverse proxy — Kong, or AWS API Gateway, or Nginx with some Lua scripts — sitting in front of a handful of backend servers. Traffic hits the gateway, the gateway checks your API key, maybe applies a rate limit, and forwards the request to an available instance. That model works fine up to maybe ten thousand requests per second. You can scale it horizontally, add more gateway instances behind a load balancer, and you're good.
And that's the diagram Daniel's describing — the one where load balancing onto servers makes sense and the API endpoint itself is just... there. An assumption.
The assumption is that the API endpoint is a single thing that receives traffic. But at millions of requests per second, the endpoint itself has to be a distributed routing fabric. You can't just put a Kong instance on a big EC2 box and call it a day. The gateway layer becomes the bottleneck.
What actually breaks first?
Configuration reloads. That's the one that catches people off guard. In a traditional Nginx or Kong setup, when you update a route or add a new backend, you have to reload the configuration. That reload might take a few hundred milliseconds — during which you're dropping connections or queueing them. At ten thousand requests per second, a two-hundred-millisecond reload drops two thousand requests. Do that a few times a day and your error budget is gone.
Two thousand requests per hiccup. That's the kind of math that keeps SREs awake.
And it gets worse. You're also dealing with hot-spotting — where one backend instance gets overloaded because the load balancing algorithm is too simple — and with cascading failures where one slow backend causes requests to pile up at the gateway. The gateway itself runs out of file descriptors or memory, and suddenly nothing works. This is where Envoy Proxy enters the picture.
Okay, let's go there. What makes Envoy different?
Envoy was originally built at Lyft — they had a microservices architecture with hundreds of services, and they needed a proxy that could handle dynamic configuration without reloads. It's now a graduated project under the CNCF, which means it's passed the same governance and adoption bar as Kubernetes. The key architectural decision Envoy made was to separate the data plane from the control plane. The data plane — that's Envoy itself — handles every request. The control plane pushes configuration to Envoy instances dynamically using something called the xDS APIs.
xDS — break that down.
It's a family of discovery services. CDS is the Cluster Discovery Service — it tells Envoy what backend clusters exist. EDS is the Endpoint Discovery Service — it tells Envoy which specific IPs and ports are in each cluster. LDS is the Listener Discovery Service — what ports and protocols to listen on. RDS is the Route Discovery Service — how to match requests to clusters. There are more, but those four are the core.
So instead of editing a config file and reloading, the control plane just... pushes new state?
In real time. A new backend spins up, the control plane discovers it, pushes an EDS update to every Envoy instance, and within milliseconds the routing tables are updated. No dropped connections, no reload pause. Envoy also does something called hot restart — when the Envoy binary itself needs to upgrade, it can hand off existing connections to the new process without dropping them.
That's the kind of thing that sounds like a party trick until you realize it's the only reason the service stays up during a deploy.
And it's not just about configuration. Envoy has sophisticated load balancing built in — consistent hashing so requests from the same user always land on the same backend, zone-aware routing so traffic stays within an availability zone when possible, least-request load balancing that sends traffic to the backend with the fewest active requests. It also does circuit breaking — if a backend starts returning errors, Envoy stops sending it traffic before the problem cascades.
Circuit breaking. That's the term I've heard but never really understood. What's actually happening?
Envoy tracks the error rate and latency for every backend. You configure thresholds — say, if more than fifty percent of requests to a backend fail in a thirty-second window, Envoy ejects that backend from the pool and stops routing to it. It'll periodically try a few requests — that's called outlier detection — and if the backend recovers, it gets added back. The whole thing happens automatically.
So you've got this proxy that never needs to reload, balances load intelligently, and cuts off sick backends before they infect everything else. That's the gateway layer. But that's still one layer. Daniel's asking about the endpoint itself — the thing at api dot openai dot com. That's not one Envoy instance, obviously.
No, and this is where the tiered architecture comes in. At the scale we're talking about — billions of requests per day, tens of thousands per second sustained — you need multiple routing layers. The first layer is typically a global load balancer — something like AWS Global Accelerator or Cloudflare. These sit at the edge, in dozens of points of presence around the world, and their job is to get traffic to the nearest regional data center as fast as possible. They're also handling DDoS mitigation at this layer — volumetric attacks get absorbed at the edge before they ever reach your infrastructure.
So the global load balancer is the bouncer at the door. What's inside?
Regional edge proxies. These are Envoy instances — or sometimes custom proxies built on similar principles — deployed in each region. Their job is authentication, rate limiting, and request shaping. They validate API keys, check quotas, and route requests to the right internal service. Behind those, you've got another layer — the internal service mesh — which routes requests to the actual model inference servers.
Three layers. Edge, gateway, mesh. Each one handling different concerns.
And the reason you separate them is that each layer has different scaling properties and failure modes. The edge layer is all about absorbing volume — it's stateless, horizontally scalable, and you can throw as many instances at it as you need. The gateway layer is where state lives — rate limit counters, API key validation caches, quota tracking. It needs to be consistent, which makes it harder to scale. The mesh layer is where the actual compute happens — GPU clusters running model inference — and that's the most expensive and least elastic part of the whole system.
So if you tried to do all three in one layer, you'd be paying GPU-cluster prices to handle DDoS traffic.
And that's the mistake a lot of companies make when they're scaling up — they don't separate concerns early enough, and then they're trying to retrofit tiered routing into a system that was designed as a monolith.
Let me ask about something Daniel mentioned — he said he's played with Kong but never seen how a public-facing API is actually managed at scale. Are there public case studies we can point to?
Stripe is the one I keep coming back to. They published a blog post a few years ago about why they built their own API gateway. They started with off-the-shelf solutions — Nginx, HAProxy, the usual suspects — and kept hitting walls. The specific problem that broke them was idempotency.
Idempotency — the thing where you can send the same request twice and it only executes once.
Right. For a payments company, idempotency isn't a nice-to-have, it's existential. If a customer hits "pay" and the request times out, they're going to hit it again. If both requests go through, you've double-charged them. Stripe's API uses idempotency keys — the client generates a unique key for each operation, and the server guarantees that operation only happens once. But off-the-shelf gateways don't understand idempotency keys. They'll happily route both requests to different backends, both backends will try to process the payment, and now you've got a mess.
So they built their own gateway that understands idempotency at the routing layer.
And that's the pattern. When your business logic starts to live in the routing layer, off-the-shelf stops working. Stripe's custom gateway handles idempotency key routing, consistent error responses across all their APIs, and request deduplication — all before the request ever reaches a payment processing service. That's not something you can configure in Kong with a plugin.
Netflix is the other one I've seen referenced. They did a big API redesign a while back.
Yes, and their insight was almost the opposite of what you'd expect. Most companies try to build a unified API — one consistent interface across all their services. Netflix tried that for years and it kept breaking. The problem was that different services have fundamentally different needs. The video streaming service needs low latency and can tolerate some inconsistency. The billing service needs strong consistency and can tolerate higher latency. The recommendation engine needs to fan out to hundreds of microservices and aggregate results. Forcing all of those through one unified API gateway meant the gateway had to be all things to all services — which made it complex, brittle, and slow.
So they stopped trying to unify.
They embraced the differences. Instead of one API gateway, they built a federation of API layers — each service boundary got its own optimized routing, its own caching strategy, its own failure handling. The client-side code was responsible for composing results from multiple service-specific APIs rather than expecting one monolithic API to do everything. It's a principle that applies directly to AI APIs.
Because different models have different capabilities, different latency profiles, different failure pattern.
Claude Opus is not the same as Claude Haiku. GPT-4 is not the same as GPT-4o-mini. Trying to route all of them through an identical gateway pipeline means you're optimizing for the average case and handling none of them well. The Netflix approach says: let each model's API boundary be different, and let the client or a smart proxy layer handle the composition.
So that's the infrastructure side. Now let's connect this back to the proxy layer discussion and what it means for the future of APIs.
Daniel's original point — the one I made, and he's running with — is that proxy layers will become normalized infrastructure, like load balancers. And when you look at how the big providers actually manage their APIs, you realize the proxy layer is already there. It's just invisible to the end user. The question is whether that proxy layer moves to the client side.
Explain that.
Right now, when you use the OpenAI API, you're hitting their edge, their gateway, their mesh. The routing intelligence lives on their side. But tools like LiteLLM flip that model — they put a proxy on your side that routes to multiple providers. You send a request to your local LiteLLM instance, and it decides whether to send it to OpenAI, Anthropic, Google, or a local model, based on cost, latency, capability, or whatever logic you configure.
So LiteLLM is doing at the client side what Envoy does inside the provider's infrastructure — routing, load balancing, failover.
It's normalizing across providers. That's the key word Daniel used — normalization. LiteLLM gives you a single API interface that works the same whether the backend is GPT-4 or Claude or Llama running locally. It translates prompts, handles authentication, maps error codes. It's a Rosetta Stone for AI APIs.
Which means the provider's API is no longer the interface. The proxy is.
That's terrifying if you're a provider and you want to own the developer relationship. But it's also inevitable — because no single provider is going to be the best at everything, and developers want flexibility. The proxy layer is an expression of that demand.
If you're Anthropic or OpenAI, what do you do about this? Do you fight the proxy layer, or do you absorb it?
You study it. Every popular proxy feature is a signal about what your API is missing. If developers are building proxies to write Claude Code output to local files, maybe Claude Code should have a native file-writing mode. If developers are using LiteLLM to route between models based on cost, maybe your API should expose cost-aware routing natively. The proxies are doing your product research for you.
That's the optimistic read. The pessimistic read is that proxies become so good that the underlying API becomes a commodity, and providers lose all pricing power.
That's a real risk, and it's why I think we'll see API providers start to build proxy-like features directly into their platforms. Anthropic could add a routing layer that automatically selects the right model for each request — Opus for complex reasoning, Haiku for simple queries, with the routing logic living inside their infrastructure. OpenAI is already doing something like this with their model selection features. The goal is to make the proxy unnecessary by being the proxy.
The infrastructure gap, basically. The proxy layer fills a gap, and eventually the API closes it.
But here's the thing — closing that gap takes time, and during that time, middleware companies like LiteLLM build moats. They get developer mindshare, they build integrations, they become the default. By the time the API catches up, the proxy is already entrenched.
What should developers actually do about all this? Given everything we've talked about — Envoy, tiered routing, Stripe's custom gateway, LiteLLM — what's the practical takeaway?
Three things. First, invest in your API gateway layer early. I don't care if you're a two-person startup — put something like Envoy or Kong in front of your services. Not because you need the scale today, but because it gives you flexibility. Want to switch from OpenAI to Anthropic? Change the gateway config, not your application code. Want to add caching or rate limiting? It's a gateway feature. The gateway is the place where you make infrastructure decisions without touching your business logic.
If you skip it, you're embedding provider-specific logic directly into your application, and unwinding that later is a rewrite.
Second, watch what proxy layers are doing. They're a leading indicator. If you see a pattern like multi-provider routing becoming popular in the LiteLLM community, expect the API providers to eventually build it in. You can either wait for that to happen, or you can adopt the proxy now and get ahead of the curve. Either way, the proxy is showing you where the puck is going.
And third?
If you're hitting rate limits or latency issues, the problem might not be the model. It might be your routing. Most developers blame the API when things get slow, but often the issue is that you're sending all your traffic to one region, or you're not using a proxy that can retry intelligently, or you're hitting a hot backend because your load balancing is naive. A multi-region or multi-provider proxy layer can solve problems that look like API problems but are actually infrastructure problems.
That third one feels like the thing nobody thinks about until they've spent a week debugging.
Because the API is a black box. When latency spikes, you can't see inside. All you know is that your requests are timing out. A good proxy layer gives you observability — it shows you which provider is slow, which region is overloaded, which model is failing. Without that visibility, you're just guessing.
The proxy layer is not a hack. It's infrastructure. Treat it that way.
That's the thesis. And I think in five years, we won't even call it a proxy layer. It'll just be part of the stack — like load balancers are today. Nobody says "I'm using a load balancer hack to distribute traffic." It's just... how you build services. The same thing will happen with AI API routing. It'll be invisible infrastructure that everyone assumes exists.
Here's my question, though. Load balancers became invisible because they solved a well-defined problem — distribute traffic across identical backends. AI API routing is messier. Different models have different capabilities, different pricing, different behavior on the same prompt. Normalizing across them isn't just a routing problem — it's a translation problem. Can that really become invisible?
I'm not sure it can, honestly. And that might be the limit of the load balancer analogy. With load balancers, the backends are interchangeable — any web server can handle any request. With AI APIs, the backends are not interchangeable. Claude Opus writes better code than Haiku. GPT-4 has different strengths than Claude. Routing between them requires understanding the content of the request and the capabilities of each model. That's not a Layer 7 routing problem — that's almost an application-layer intelligence problem.
The proxy layer for AI might always be smarter — and more visible — than a traditional load balancer.
Or it might get absorbed into the models themselves. Imagine a future where you don't choose a model at all — you send a prompt to an API, and the API's internal routing fabric decides which model or combination of models to use, based on the prompt, your budget, your latency requirements, and the current load on each model cluster. The routing intelligence becomes part of the platform, not something you configure.
That's the Netflix model applied to AI — embrace the differences between models, but hide that complexity behind a smart routing layer that the provider owns.
That's the bet the big providers are making. They want to be the platform, not the model. The model is a feature. The platform is the moat.
Which brings us back to Daniel's original question about how these APIs are actually managed at scale. The answer, I think, is that they're managed through layers of proxies — Envoy at the gateway, global load balancers at the edge, service meshes internally — and the whole thing is a distributed system that most developers never see. But understanding it matters, because the same patterns are showing up on the client side with tools like LiteLLM.
The line between "provider infrastructure" and "client infrastructure" is blurring. What the provider does internally today, the client will do externally tomorrow. The proxy layer is just that pattern moving up the stack.
Given all this, what should you actually do about it? Here are three takeaways.
One — if you're building an AI-powered product, invest in your API gateway layer early. Tools like Envoy or Kong aren't just for big companies. They give you flexibility to change providers, add middleware, and handle scale without rewriting your application.
Two — watch what proxy layers are doing. They're a leading indicator of where APIs are going. If a proxy pattern becomes popular, expect the API provider to eventually build it in. You can wait or you can adopt early — but either way, pay attention.
Three — for developers hitting rate limits or latency issues, the problem might not be the model. It might be your routing. Consider a multi-region or multi-provider proxy layer before you blame the API.
The proxy layer is not a hack. It's infrastructure. Treat it that way.
Here's the open question I keep coming back to. As AI APIs mature, does the proxy layer become a standard part of the stack — like load balancers — or do the APIs absorb these functions and make proxies obsolete? I could see it going either way.
I think it's both. The simple routing and load balancing functions get absorbed into the platform — that's already happening. But the translation layer — the thing that makes Claude and GPT and Llama all look the same to your application — that might persist, because the providers have no incentive to make their APIs interchangeable with competitors. The proxy layer thrives on the gaps between providers, and those gaps aren't going away.
The line between API and infrastructure keeps blurring. The next generation of developers might never think about routing at all — it'll just be part of the platform. But for now, understanding what's happening behind the curtain is the difference between building something that works at scale and something that falls over the first time you get real traffic.
That's the thing Daniel was really asking — not just "what tooling exists," but "how do I think about this layer of the stack." The answer is: think of it as a distributed system in its own right, with its own scaling properties, failure pattern, and design patterns. It's not plumbing. It's architecture.
Thanks to our producer Hilbert Flumingtop for keeping this show running. If you've got a weird prompt about infrastructure, APIs, or the future of AI tooling, send it to us — my weird prompts dot com. We read every one.
This has been My Weird Prompts. I'm Herman Poppleberry.
I'm Corn. We'll be back soon.
