Episode 6334 custom

Why Fast Backups Trigger Rate Limits

Listing files on Google Drive costs 100 quota units per call, while downloading costs 200, making folder traversals surprisingly expensive.

Oct 28 28:58 Transcript

With

0:00

cloud-computing backups-and-archiving

Episode Details

Episode ID: MWP-6536
Published: Oct 28
Duration: 28:58
Audio: Direct link
Pipeline: V5.25
TTS Engine: chatterbox-regular
Script Writing Agent: GPT-6.1 Sol
Topics: cloud-computing backups-and-archiving

AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.

The prompt

The prompt that started this episode, as sent in (production notes for the hosts removed).

As agentic AI tools get more capable, they're driving CLIs and APIs on ordinary people's behalf, and a lot more people are about to run into rate limiting. It's a legitimate mechanism against abuse, but these days it seems just as often to catch regular people who've discovered CLIs and are using them with AI agents, and they get treated like spammers. My own example: backing up my Google Workspace with a CLI tool.
Start with the fundamentals, because many listeners will be new to this. From the defender's side: how does rate limiting actually work, what are the first things to know about it, and what trips it? Then the other side: retry logic, backoff and the basics of being a well-behaved client, so you know what you're up against.
The real question: are we stuck playing cat and mouse forever, or is anyone working on a way for legitimate users and their agents to be easy to tell apart from people hammering APIs for illegitimate reasons? Are there emerging ideas or best practices for people running agents for their own purposes?

Automated backups can stumble into rate limits even when saving your own files, exposing the tension between authentication and infrastructure capacity. When agents or tools like rclone interact with APIs like Google Drive, an HTTP 429 or quota-related 403 does not necessarily indicate malicious intent. Legitimate operations easily outpace human browsing, and resource costs depend heavily on weighted methods—such as directory listings costing far more quota units than inspecting metadata—rather than raw download sizes. Furthermore, quotas exist across multiple layers, including per-user, project, and unadvertised backend thresholds. Effective client design demands classifying errors accurately, since blind retries or unmanaged concurrency waste capacity without solving persistent configuration boundaries.

Sources

What the research for this episode read before the script was written. Primary sources first.

Mentions

Anchor Browser Browser infrastructure provider for agents
Browserbase Browser infrastructure provider for agents
ChatGPT agent OpenAI agent example in signed agent list
Cloudflare Web infrastructure and security company
Google Drive API Google Drive quota and rate limits
Google Workspace CLI (gws) Unofficial Google Workspace command line tool
Goose Open-source AI agent from Block
IETF RateLimit headers draft Proposed standard for quota headers
rclone Command-line program to sync files to cloud
Web Bot Auth Signed agent identity via HTTP Message Signatures

Downloads

Episode Audio

Download the full episode as an MP3 file

Download MP3

Transcript (TXT)

Plain text transcript file

Episode Book (PDF)

The episode's record — date, duration, models, sources — with the full transcript

#6334: Why Fast Backups Trigger Rate Limits

Your backup stops halfway through, the terminal says too many requests, and suddenly saving your own files looks suspicious. Daniel ran into this backing up Google Workspace with a command line tool. His question is what happens as AI agents put that kind of automation within everybody's reach. How does the service decide you're asking too fast, what does a decent client do when it gets refused, and can legitimate agents identify themselves without looking like spammers?

The first distinction matters immediately. An HTTP four twenty-nine means the service is pushing back on traffic. It does not necessarily mean anyone has decided you're malicious. You might be competing with other perfectly legitimate work, or hitting a limit your dashboard doesn't display.

Innocent, but still not backed up.

Right. The user sees a failed job. The defender sees a stream of operations against shared infrastructure. Neither gets to inspect the other's intentions directly.

Also, Herman, I predict you're about to admire the elegance of the limiter before we've recovered a single document.

I admire an elegant limiter that tells you what it limited. A mysterious refusal with a cheerful retry loop gets much less admiration.

Your standards are rising.

Living with your software reviews has consequences.

Rate limiting starts with a counter and a bucket to count against. That bucket might belong to an account, an IP address, an application project, or some combination. Each operation consumes capacity. Once the allowed capacity is exhausted, the server delays or rejects more work until capacity returns.

The uncomfortable bit is that ordinary use has changed shape. Reading documents through a browser involves pauses. A backup doesn't stop to consider a paragraph. It lists, checks, downloads, repeats. Give that job to an agent and the person issuing it still feels like they're making one request.

Whereas the API may see thousands. Older descriptions of attack traffic sometimes contrast a compromised machine issuing more than twenty web requests a second with much lower legitimate rates. That's an illustration, not a universal threshold. A legitimate command line client can exceed that pace quite comfortably.

Twenty requests a second is now somebody being tidy.

The speed doesn't establish intent. More importantly, honest traffic can still exhaust a service. If everybody backed up a large account at once, the infrastructure would have a capacity problem even if every request came with a handwritten apology.

I accept the need to ration. I object when the ration depends on a secret unit and the refusal says something unrelated. A backup is especially irritating because the files already belong to you. You're not trying to collect somebody else's website.

Ownership establishes why you should be allowed access. It doesn't make enumeration, permission checks, database queries, or outgoing bandwidth free. That's why authentication and capacity controls coexist.

Fair. Though the interface encourages the confusion. You sign in, approve access, and the tool says it's ready. Nobody says that ready means ready to discover a second permission structure made of counters.

Would you prefer a slower backup that finishes overnight, Corn, or a faster one that occasionally pauses unpredictably?

Overnight, if it gives an honest completion estimate and resumes properly. Unpredictable pauses are tolerable. Unpredictable restarts aren't. I don't want an agent spending the night enthusiastically rediscovering the first folder.

That distinction is going to matter when we get to retries. First, the counters themselves can explain why a tool that barely seems busy is already in trouble.

A fixed window limiter counts work within a defined interval. A sliding window tracks a moving interval, avoiding some boundary effects. A token bucket allows a limited burst, then replenishes capacity at a steady rate. These designs behave differently even when their advertised average allowance sounds identical.

Meaning a minute's allowance doesn't necessarily mean I can spend it in the first second.

Correct. A service can permit a sustained rate while rejecting an abrupt burst. It can also maintain several counters simultaneously. Your account has one allowance, the application's project has another, and the backend may enforce additional protections.

That creates a nasty household problem. My job slows down because your job is busy, but my error message makes it sound personal.

Shared project quotas really can do that. The request is authenticated as yours, yet it consumes a pool other users of the same application also draw from. A shared network can create a different version through IP limits. Separate identities don't guarantee separate capacity.

The technical term is apparently togetherness.

Drive makes the layered accounting unusually concrete. For projects under the quota scheme introduced in May, the published project allowance is one million quota units per minute. The per-user allowance within that project is three hundred twenty-five thousand per minute.

Those sound enormous until you tell me what a unit buys.

Listing files costs one hundred units per API call. A metadata get costs five. The download method costs two hundred. Those are method weights, not a count of files or a direct measure of bytes.

Looking around costs twenty times as much as looking at something.

Per call, yes. A listing call can return a page containing multiple files, so you mustn't translate that into one hundred units per file. Page size, folder structure, requested fields, and the tool's traversal strategy all affect the work.

That correction changes the mental picture. A well-designed listing can cover plenty of files, while a tool that repeatedly walks folders spends capacity proving it has folders. The backup hasn't failed because the documents are enormous. It may have failed while assembling the shopping list.

A method that updates a file costs fifty units, too. Read versus write is not a sufficient guide to cost. Services assign weights to reflect their own resource model, and that model won't necessarily match what feels expensive to you.

I would have guessed downloading dominates everything.

Bytes matter separately. The published scheme includes one terabyte of egress per day per project. There's also a daily billing threshold of four hundred million quota units that the documentation says cannot be raised. A billing threshold and a short-term rate allowance are different concepts.

More counters. Fewer documents.

Older projects can retain older quotas, which is why comparing two people's screenshots without checking their project history can be misleading. You can both be describing your setup accurately and still have different allowances.

This is where an agent can make confidence actively unhelpful. It sees a large published quota, observes much less activity, and concludes the service must be broken. Then it increases concurrency to make progress.

Or treats the error as an instruction to repeat the operation without changing anything. The published quota is only part of the picture. Drive explicitly says additional backend rate checks can generate a four twenty-nine response.

The dashboard can be green while the backup is stationary.

Yes. A dashboard for a project allowance doesn't promise visibility into every backend constraint. Local hotspots, shared resources, or protective checks can still matter. I don't know which internal check fired in any particular user's case, and a quota chart alone can't tell us.

That uncertainty should be in the error response, rather than something users reconstruct from three evenings of reducing settings.

Let me ask you something. If the service disclosed every protective threshold, would you consider that an unconditional improvement?

Not every abuse threshold. An attacker shouldn't receive a detailed map of how to stay invisible. But an authenticated customer should be told whether the next useful action is waiting, reducing concurrency, changing permissions, or asking for a different allowance. That's operational information, not necessarily the burglar alarm's wiring diagram.

There's a genuine balance there. Detailed detection logic can help attackers adapt, while vague feedback makes legitimate clients waste capacity. A service can disclose the category and expected recovery behavior without publishing its entire abuse model.

Especially when the HTTP number is already doing too many jobs.

Drive quota errors can arrive as four oh three with a user rate limit message. Additional backend throttling can arrive as four twenty-nine. But four oh three also covers unrelated permission problems. The client needs the structured reason, not just the status number.

We should be exact about waiting here. A per-minute quota can recover after time passes. Calling every quota error permanent would be as misleading as calling every one temporary.

I was about to compress that too far. A minute allowance can replenish. A daily allowance needs a different timescale. An insufficient authentication scope won't improve with waiting at all. A project capacity problem may require configuration or a quota adjustment rather than another short retry.

Sleeping doesn't grant permissions.

That's why the instruction to use exponential backoff is incomplete without error classification. Backoff helps when time or reduced pressure can change the outcome. It doesn't manufacture authorization or remove a hard ceiling.

People using rclone have reported excessive user rate limit errors during fast listing of shared drives, including cases where their visible project quota wasn't under pressure. That's a report of a troublesome pattern, not proof of exactly which hidden protection fired.

Fast listing seems an especially cruel label to leave enabled while nothing lists.

Another user reported that a sync completed only after reducing both transfers and metadata checkers to one apiece. The distinction matters. Download workers aren't the only source of traffic. Parallel checking can generate plenty of requests before the transfer display gets interesting.

I can already hear your objection to the apparent fix. One worker succeeding doesn't prove one worker is the universal answer.

You anticipated my objection before I could enjoy making it.

You've been looking at the word universal suspiciously all morning.

Different drives, projects, and operations have different bottlenecks. Still, those reports establish something useful. More concurrency can reduce useful throughput once it provokes rejection and retries.

The agent needs to count completed work, then. A terminal full of activity is a very poor substitute. I have seen your installations produce tremendous activity, Herman.

Some of that activity was diagnostic.

The backup remained a thought experiment.

That is why I now distinguish a successful command from a successful outcome. The Google Workspace CLI, gws, also deserves that distinction. Its own README says it is not an officially supported Google product, despite the name and its positioning for humans and agents.

An important sentence to read before delegating confidence.

It's pre-version-one software with a command surface generated from Google's Discovery Service. That helps it cover changing APIs, but it doesn't establish that every agent-generated workflow has sensible request economics. Dynamic command discovery and careful scheduling are separate achievements.

A tool can know every available operation and still choose an expensive way to do the job.

Now consider what happens when several layers all try to help. The library retries. The CLI retries. The agent sees no progress and reruns the CLI. A single intention has become overlapping attempts, possibly against the same shared allowance.

Persistence with multiplication.

There needs to be one coherent retry budget, or at least coordination between those layers. Otherwise the visible pause at one layer conceals pressure from another.

Here's a voicemail from a listener out there in the wild.
Tilda: Hey Corn, hey Herman, this is Tilda, I'm calling from the parking lot of the Piggly Wiggly on Highway Seventeen, I've got the truck running because it's already ninety-four degrees out here. I listen every week while I'm doing my grocery run, and today's episode about rate limiting and backup scripts got me thinking about something. I don't run any of that agent stuff myself, but my nephew set me up with one of those command line tools to back up my church choir photos to the cloud, and after about an hour it just quit on me and said I'd hit some kind of limit. So my question is, when it tells you that, is it telling you to wait a little while or to wait a long while? Because I waited twenty minutes like it said and it quit again right away, and I've got four hundred photos of the Easter cantata that still aren't backed up.

Tilda, that immediate second stop could mean the next run recreated the same burst, or that a different allowance hadn't recovered. The useful missing information is what limit was reached and whether the job resumed its progress. Elapsed waiting time alone doesn't tell you either.

Resuming matters enormously. If every restart repeats the inventory, the tool can spend the renewed allowance reaching the exact point where it stopped last time. Patient users become unpaid participants in a very slow loop.

Checkpointing changes that. Preserve completed work and the position in a long operation, where the API supports it. Then a pause can actually reduce outstanding work instead of resetting the task.

Also, there's a difference between retrying the rejected request and replaying the whole workflow. An agent saying I'll try again may mean either, and the user rarely gets asked which one they'd prefer.

Google recommends truncated exponential backoff for transient limits. The early waits are about one second, then two, then four, with a fresh random addition each time. The intervals grow rather than staying fixed.

A refusal gets progressively more personal space.

The random addition is up to a second in that example. Eventually the delay reaches a cap, typically thirty-two or sixty-four seconds. Truncated means it stops growing indefinitely, not that it stops retrying automatically.

Which leaves the slightly awkward question of when it does stop.

A separate limit has to answer that. Maximum attempts, total elapsed time, or a job deadline. The documentation itself says clients should eventually be prevented from retrying indefinitely. Otherwise capped backoff can become a permanent background load.

I'd want an unattended job to stop with a useful account of what remains. Not silently surrender, and not spend three days interpreting no as encouragement.

You like unattended automation until it demonstrates your own resistance to getting up.

I stop when the job is impossible.

You also stop when the job is possible.

A broader compatibility range.

Jitter addresses a different problem from growing delays. Imagine many clients hit the same limit and all wait exactly the same interval. They return together, overwhelm it together, then retreat together. Randomizing the waits spreads the returning traffic.

A pause can be individually polite and collectively disastrous.

It can. There's another distinction I don't want to lose... backoff governs retries after trouble. Pacing governs normal traffic before trouble. A well-behaved client shouldn't require a stream of refusals to maintain a reasonable operating rate.

Reducing retries while keeping the same aggressive batch starts doesn't really learn anything. It just adds ceremonial waiting between collisions.

A feedback controller can lower concurrency or request rate when rejection rises, then cautiously explore more capacity. That is closer to discovering an acceptable rate than running the same burst forever.

Though we shouldn't casually retry every operation either. A failed response doesn't always mean the operation failed to happen. Reads are generally easier here. Repeating a write can create duplicate work unless the API offers a way to make retries safe.

That distinction matters for agents doing more than backups. A request may have completed server-side before the connection failed. Retrying needs to account for operation semantics, not merely transport errors.

The model doesn't need to improvise all of this every time. I'd rather the tool own the scheduling and resumability, and the agent explain what the tool is doing. Otherwise a conversational interface becomes a new place to implement network congestion control badly.

Would better instructions from the server satisfy you, or would you still want an independent client-side cap?

Both. The server can describe its allowance, but my own cap can protect my bandwidth, budget, and other work. More importantly, the server's estimate might be stale by the time another worker spends the remaining capacity. An advertised allowance shouldn't be treated as a reservation.

The proposed RateLimit headers explicitly make that distinction. Available quota doesn't grant requests. Conditions can change, and other clients may share the same bucket.

There's active standards work at the IETF to give clients structured information before they repeatedly hit errors. The draft defines RateLimit-Policy and RateLimit fields for advertised policies and current limits. It remains a draft, rather than a promise that every API already speaks this language.

That would let the tool distinguish a replenishing allowance from a vague instruction to become less troublesome.

It can communicate the relevant partition, the effective window, and available quota. Retry-After can provide explicit pushback. The important change is that a client gets something it can schedule against, rather than inferring the rules entirely from rejection.

Provided the server actually publishes it.

Provided it publishes it, the client understands it, and both interpret the units consistently. A bare remaining balance means little if nobody knows whether the next operation costs one unit or two hundred.

The draft also separates reasons for trouble, doesn't it? That's the part I'd value most. I can tolerate a slower service much more easily than a service that won't say whether it thinks I'm unauthorized, expensive, or suspicious.

It defines problem types for quota exceeded, temporarily reduced capacity, and abnormal usage detected. Those convey meaningfully different situations. The last is much closer to an abuse-related signal than an ordinary exhausted allowance.

Finally, separate messages for the queue and the accusation.

Even then, abnormal usage doesn't establish malicious intent. It means the pattern triggered that category of protection. A legitimate application might need to change its behavior, while the operator may need to investigate a false positive.

Hm.

There's still a missing half. These headers tell my agent what the server wants. They don't tell the server that my agent belongs to somebody with a legitimate purpose. An attacker can read scheduling information too.

That is where Web Bot Auth enters. It uses HTTP Message Signatures to let automated clients cryptographically identify their requests. Instead of trusting a self-declared User-Agent name, the server verifies a signature against a public key associated with the agent.

Writing respectable crawler on the envelope has had a long run.

User-Agent strings are easily copied. IP verification has problems with shared hosting, VPNs, and privacy proxies. Per-site bearer credentials become cumbersome across large numbers of sites. Signatures offer a different basis for recognizing the sender.

Recognition still doesn't establish virtue. A signed agent can be signed and annoying.

Authentication identifies the signer. Authorization decides what that signer may do. Rate limiting still governs consumption. You can trust that a request came from a particular platform while refusing it because it exceeds capacity or violates your policy.

The signature carries a validity window and a key identifier. A Signature-Agent field points toward a directory where verification keys can be found. The signature covers the target authority, binding the proof to where the request is aimed. Those details make it stronger than a reusable label.

You're getting that look again. You like the cryptography.

I like replacing a guess with a verifiable statement.

Now verify that I get admitted.

That part remains policy. Cloudflare distinguishes signed agents directed by end users from conventional verified bots. Its initial examples include ChatGPT agent and Goose from Block, alongside browser infrastructure providers Browserbase and Anchor Browser.

Goose complicates the easy claim that everything here must run inside a giant platform. It's an open agent ecosystem. We shouldn't confuse who operates a particular request with the product name on the list.

Agreed... no, more specifically, recognition of Goose does not automatically mean every independent deployment has the same recognized identity. The protocol's ability to carry a signature and a site's willingness to trust that identity are separate questions.

Anyone can make a key. Not everyone can make a key anybody cares about.

That's the structural gap. An individual self-running a CLI can, in principle, implement signing. But that doesn't automatically earn directory inclusion, recognition by a security provider, or favorable treatment from websites. Cloudflare's enrollment involves an application process.

I'd call that a potential moat rather than a mathematical lockout. The cryptography isn't closed to individuals. The distribution of trust may be.

That is the distinction I would defend. A recognized platform identity can arrive with reputation and established relationships. A personal identity starts without those advantages. How much access it gets is an adoption and governance problem.

Your question, then. Would you route your personal backup through a recognized platform if that made it work reliably?

For some tasks, yes. For sensitive data, I'd want to understand the custody and permissions first. Reliability can tempt people into adding an intermediary they otherwise wouldn't choose.

That's my worry too. Self-hosting gives you control over your tool, but the website may know less about you than about the commercial intermediary. Privacy-preserving network choices can also make IP reputation less useful. The person trying to reduce disclosure can end up being asked to disclose more.

Google's experimental adoption shows the transition isn't complete even for major operators. It says only a subset of its agent requests are signed and tells site operators to fall back to IP-based verification.

New identity, old checkpoint.

Both approaches require participation on the other side. A client can't force an API to publish useful quota information, and a signature can't force a site to recognize its signer. Useful standards don't remove the defender's authority to say no.

Nor should they remove it. I'd settle for a no that stops wasting everybody's afternoon. A retry loop continuing after its user has given up isn't autonomy worth celebrating.

Yes, I agree there. I keep the rejected responses in a squared notebook. Just the time and the number. This month I was downloading drainage records from the council portal for my flat. Martin Bell in the records office said the export would be ready that afternoon. The link arrived three days later. My little downloader got a four twenty-nine and started again every five seconds. I had written that part myself.

Five seconds leaves plenty of room in the notebook.

I use narrow columns. Herman, your distinction between a pause and a restart was the part that mattered. Mine restarted the listing before it tried the download. Every attempt did the same preliminary work. I left it running while I went shopping. It had made no progress when I got home. Take that chair scrape out before the word preliminary. The notebook is with Martin now, at the records counter. He asked for the error times. I wasn't copying them out twice.

Repeating the listing explains why the downloader could remain busy without getting closer to the file.

Five seconds seemed reasonable at the time. I wasn't asking for the whole borough. My brother-in-law maintains booking software and says people blame the limiter because the waiting is visible, while the repeated work isn't. He says the first retry often gets written carefully and the surrounding restart gets forgotten. He would say that. He's still trying to explain a booking system that sent me the same cancellation all afternoon. Martin eventually sent the records as an attachment. That took twelve minutes once he had my email in front of him. He had to check the drawings before he sent them.

...Whether an ordinary user can eventually get that kind of clear recovery without moving into a platform's identity system is what I'll be watching. Better quota disclosure and verifiable signatures can reduce guessing. They only broaden access if services actually support them and recognize identities beyond the established intermediaries.

Our thanks to Hilbert Flumingtop for producing the show.

If you want more of this, try episode five, Fine-Tuning ASR For Maximal Usability; episode nine, Benchmarking Custom ASR Tools - Beyond The WER; and episode sixteen, On Deepfakes, SynthID, And AI Watermarking. This has been My Weird Prompts. No question too obscure, no rabbit hole too deep.

If you enjoyed this one, leave us a review wherever you listen. It helps other curious people find the show.

See you tomorrow.

This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error

Why Fast Backups Trigger Rate Limits

Sources

Mentions

Downloads

Recommended Listening

You Might Also Like

#6334: Why Fast Backups Trigger Rate Limits