So Daniel writes in with a build question, and this one has a very specific smell to it. He wants a unified virtual filesystem on his home server. The consumer is AI agents, not him clicking through folders. The server runs an MCP gateway, so the agents are what actually touch storage. He has some shared drives, some S3 buckets across different providers, and MinIO is already running on the box. What he wants is one virtual remote where everything shows up in a single tree, whichever cloud it physically lives on. MinIO handles getting files onto the server. The open question is how they get from there onto the external clouds, and how the whole thing gets deployed. He says skip the deep mechanics, make this a straight how would you actually do this episode. Three things he wants decided. One, does MinIO front the whole thing, or sit beside it as just another backend. Two, what is the union layer, rclone union or something better for an always-on server. Three, how do agents reach the finished namespace, S3 API, a mounted path, something else, given an agent wants a stable path it can hand to a tool, and it cannot click reauthorize when a shared drive token expires. Parts list, deployment order, and what I would do differently if I were him.
This is a genuinely good project. Most people build unified storage so a human can browse one folder tree and feel organized. Daniel is building it so an AI agent can reach into four different clouds without caring which one holds the file. That is a completely different set of constraints. Stability, programmatic access, zero interaction reauthorization. Not pretty folder icons.
Right, because the agent is not going to sit there and wait while you dig out your phone and tap allow on a Google consent screen at three in the morning. It needs a path that works every single time. Which is the actual hard part of this build. Not the union layer. The token problem.
And that is exactly why the three questions he asked are not three separate questions. They are one question in three parts. The union layer choice constrains the access protocol. The access protocol constrains whether MinIO fronts the whole thing or sits beside it. You cannot decide them independently. You have to think of it as a dependency chain.
So let us actually build this thing. But first, let us understand what we are optimizing for.
Here is the frame. The consumer of this filesystem is an AI agent talking to an MCP gateway. That means the design has to optimize for three things. One, a stable endpoint the agent can hand to a tool and trust to work tomorrow. Two, programmatic access, meaning the agent speaks an API, not a file browser. Three, zero interaction reauthorization. When a token expires, something has to refresh it silently, because the agent cannot click a button and Daniel is asleep.
So the human UX stuff, browseability, thumbnails, drag and drop, none of that matters. What matters is whether a tool call that says read this object returns the bytes or hangs for six minutes.
And that reframes the whole build. The headline architecture is actually pretty clean. MinIO as the front-end S3 endpoint. Rclone union as the merging layer behind it. Agents talking S3 API. But the details of how rclone mounts and serves matter more than the headline choice. The failure modes are where this thing lives or dies.
And before we dive in, let me just say the obvious thing. Could Daniel buy a product that does this? There are commercial cloud aggregators, there are tools that sync everything into one bucket. But none of them give him the control he wants, and most of them want a monthly subscription. For a home server running an MCP gateway, this is a legitimate DIY project. The parts are free, the complexity is real but manageable, and the result is something he fully controls. So yes, worth building.
Agreed. And the parts list is refreshingly short. This is mostly software. The hardware he already has, the home server, is the platform. So let me lay out the architecture first, then we get into decisions.
The first decision is where MinIO sits, and that choice ripples through everything else.
So let us talk about MinIO. Daniel says MinIO is already running on the box. And there is a temptation to say, well, it is already there, so it should front the whole thing. That is a misconception. MinIO being already running is not a reason for it to be the front end. It is a reason to consider it. The actual decision should be based on what MinIO is good at.
And what is MinIO good at?
MinIO is an object store. It speaks the S3 protocol natively. It handles bucket semantics, object listing, authentication, versioning, lifecycle rules, all of that. What MinIO does not do is merge multiple backends into one namespace. It is not a union filesystem. It is a place to put objects. So if MinIO is going to front the whole thing, the union has to happen before MinIO sees the data. Rclone union feeds MinIO, and MinIO presents the merged view to agents.
So MinIO is the pretty storefront, but rclone is the warehouse behind it doing the actual merging.
Correct. And here is the alternative. MinIO could sit beside the other clouds as just another backend. Rclone can treat any S3-compatible endpoint as a remote. So MinIO would be one remote among several, Google Drive another, Backblaze B2 another, and rclone union would merge them all directly. No fronting.
And why would that be worse?
Because then the agents would need to know which backend holds which file. That defeats the entire single tree requirement. The union layer has to be the thing agents see. If MinIO is just another backend, the union is what agents see, and the union is exposed directly, probably through rclone serve S3. So MinIO becomes redundant in that scenario. It is just another bucket, not a front end.
So the real choice is between two architectures. One, rclone union merges everything, then rclone serve S3 exposes the union directly to agents, no MinIO fronting. Two, rclone union merges everything, feeds the merged view into MinIO, and MinIO exposes the union as an S3 endpoint.
And both are legitimate. Let me talk through the tradeoffs, because this is where the decision actually gets made.
Before you do, let me ask the question I think Daniel is actually asking. He said MinIO is already running. He probably assumes it is doing something useful. So what does MinIO fronting actually buy him?
A few things. One, MinIO gives you a polished S3 endpoint with bucket management, access keys, policies, all the stuff an agent tool can consume without thinking. Two, MinIO can do server-side operations that rclone serve S3 does not do, like multipart upload orchestration, object locking, encryption at rest. Three, if Daniel ever wants to add a second consumer, like a backup tool or a second agent, MinIO gives him a real multi-tenant S3 service with its own auth layer. Rclone serve S3 is more of a thin translation layer. It works, but it is not a full S3 implementation.
And what does rclone serve S3 buy him that MinIO does not?
Simplicity. Fewer moving parts. If rclone union is already merging the remotes, rclone serve S3 can expose that union directly. No MinIO in the loop. One process instead of two. Less to go wrong. And rclone serve S3 handles the S3 protocol well enough for agents. It supports bucket listing, object get and put, multipart uploads. It is not as feature complete as MinIO, but for an agent that mostly reads and writes objects, it is plenty.
So the question is whether the extra polish of MinIO is worth the extra complexity of running it in front of rclone.
And here is my honest take. For this build, I would skip MinIO as the front end and use rclone serve S3 directly on the union. Not because MinIO is bad. Because rclone serve S3 does the job with less overhead, and the agent does not care about the features MinIO adds. The agent wants a stable S3 endpoint. Rclone serve S3 gives it that. Adding MinIO in front means running two services where one would do.
But Daniel said MinIO is already running. So is he supposed to just ignore it?
No. MinIO can still be useful. Here is a nice middle path. MinIO stays running as a local staging area. Agents write files to MinIO first, because it is fast and local. Then a separate rclone sync or rclone copy job pushes those files from MinIO out to the external clouds on a schedule. That gives Daniel the best of both. Fast local writes for agents, and a reliable sync layer to the clouds. MinIO is not the front end for the unified view, but it is not wasted either.
That is actually a cleaner design. Agents write locally to MinIO, rclone syncs to the clouds, and rclone union plus serve S3 exposes the read path. MinIO is the ingest buffer, not the front end.
And that is what I would do differently if I were Daniel. He said MinIO handles getting files onto the server. So he is already thinking of MinIO as the ingest point. The mistake would be to also make it the front end, because then rclone union has to feed MinIO, and MinIO has to serve the union, and you have two processes in the critical path for reads. Keep MinIO as the write buffer, let rclone handle the read path, and the whole thing gets simpler.
So decision one is settled. MinIO sits beside the architecture as the local ingest buffer, not in front of it. The front end for agents is rclone serve S3 on the union remote. Now decision two, the union layer.
And the answer here is rclone union. Not because there are no alternatives, but because the alternatives do not fit this problem. Let me explain why.
What are the alternatives?
The big one people reach for is mergerfs. Mergerfs is a FUSE filesystem that merges multiple directories into one mount point. It is excellent for local disks. But it works at the filesystem level, not the remote level. It does not know anything about S3 eventual consistency, or OAuth token refresh, or cloud API rate limits. Rclone union works at the remote level. It sits on top of rclone's remote abstraction, which already handles all the cloud-specific weirdness. So when a Google Drive token expires, rclone's drive backend refreshes it silently, and the union never even notices.
That is the key difference. Rclone union inherits all the cloud handling from rclone's backends. Mergerfs would be sitting on top of already-mounted remotes, and each mount would have to handle its own token problems, and the union would just see a filesystem that sometimes hangs.
And there is another alternative, which is rclone's own combine backend. Combine is newer, and it merges remotes in a static way. But union is more mature and has better policies for handling conflicts. For this use case, union is the right call.
Let us talk about those policies, because this is where the always-on server requirement bites.
Right. Rclone union supports multiple upstreams with configurable policies. The two that matter for an always-on server are the read policy and the cache policy. The read policy determines which upstream rclone checks first when an agent asks for a file. The cache policy determines what happens when a backend is temporarily unreachable.
Walk me through the cache policies.
There are three main ones. The default is off, which means no caching of directory listings. Every read goes straight to the upstreams. The second is lru, least recently used, which caches directory entries and evicts the oldest when the cache fills. The third is ephemeral, which caches directory entries but expires them quickly, so the union does not serve stale data. For an always-on server with agents reading files, I would use lru with a modest cache size. It gives you resilience when a backend is slow, without serving stale data for too long.
And what happens when a backend is completely unreachable? Does the union hang, or does it fail fast?
This is the critical question, and it is where most union filesystems fall over. Rclone union has a policy called action policy, and the relevant setting is what happens on a backend error. The default is to return the error to the caller. So if an agent asks for a file that lives on a backend that is down, the agent gets an error, not a hang. That is what you want. A hang is the worst possible outcome for an agent, because the agent has no idea whether to retry or give up. An error is actionable.
So the union layer is not just about merging folders. It is about how failures propagate. That is the thing people get wrong. They think union means combine everything and forget about it. But the failure handling is the whole ballgame.
And that is exactly the misconception we should flag. The union layer is not just about merging folders. It is about what happens when one of the folders disappears. For an always-on server, that is the critical design consideration.
Let us get concrete. Daniel has some shared drives and some S3 buckets. How does the union remote actually get configured?
First, he configures each remote individually in rclone. So a Google Drive remote, a Backblaze B2 remote, maybe a Wasabi remote, and the local MinIO remote. Each one gets its own auth. Google Drive uses OAuth with a refresh token. Backblaze B2 uses a key ID and application key. MinIO uses an access key and secret key. Rclone stores all of this in its config file, and it handles token refresh automatically for backends that support refresh tokens, which Google Drive does.
The token expiry problem is solved at the per-remote level, before the union even exists.
Correct. And that is the beauty of rclone. The union remote does not need to know anything about auth. It just sees three or four upstreams that rclone already knows how to talk to. The union config is just a list of remotes and policies.
Walk me through the actual config command.
He runs rclone config, creates a new remote of type union, and then specifies the upstreams. Something like drive remote colon, b2 remote colon, minio remote colon. Then he sets the read policy. For an agent that mostly reads, I would use a policy like newest or lru, depending on whether he wants the most recent version or the fastest response. Then he sets the cache policy to lru with a cache time of maybe five minutes. That gives some resilience without serving stale data for too long.
What about writes? Where do new files go?
The write policy. Rclone union supports several. The default is to write to the first upstream in the list. But for Daniel's setup, where agents write to MinIO first and rclone syncs out, the union remote is mostly read-only from the agent's perspective. So the write policy matters less. But if an agent does write through the union, he should set the write policy to the MinIO remote, so new files land locally and get synced out later.
That is a clean pattern. Agents write to the union, the union writes to MinIO, and a separate sync job pushes MinIO out to the clouds. Reads go through the union, which pulls from whichever cloud actually holds the file.
Here is the thing about the sync job. It does not have to be fancy. A simple cron job running rclone sync minio remote colon bucket slash external cloud remote colon bucket every fifteen minutes is enough. Or rclone copy if he wants one-way only. The point is that the union gives agents a single read path, and the sync job gives Daniel a single write path to the clouds. Two separate concerns, two separate mechanisms.
Decision two is settled. Rclone union, with lru cache policy, fail-fast on backend errors, and MinIO as the write target. Now the third decision, how agents reach the finished namespace.
This is where the design gets interesting. Because there are two main options, and they look similar but behave very differently.
The S3 API versus the mounted path.
Right. Option one is rclone serve S3. This command takes any rclone remote, including a union, and exposes it as an S3-compatible endpoint. Agents talk to it using the S3 protocol, which means they use bucket names, object keys, and standard S3 operations like get object and put object. Option two is rclone mount. This mounts the union as a POSIX filesystem, so agents see a regular directory tree and use standard file I/O.
Daniel asked specifically about the stable path requirement. An agent wants a path it can hand to a tool and trust to work tomorrow. Which option gives it that?
Both can, but in different ways. With rclone serve S3, the agent hands the tool an S3 endpoint, a bucket name, and an object key. Those are stable as long as the serve process stays up and the credentials do not change. The credentials are the part that matters. Rclone serve S3 can be configured with static access keys, so the agent's credentials never expire. That is a huge advantage. The agent does not need to reauthorize, ever.
And with rclone mount?
The agent hands the tool a file path, like slash mnt slash union slash bucket slash object. That path is stable as long as the mount process stays alive. But mounts are stateful. They depend on the FUSE mount staying healthy. If the mount process crashes, the path disappears. If the mount gets into a weird cache state, the agent might read a stale file. And mounts introduce caching semantics that can confuse an agent expecting immediate consistency. An agent writes a file, then reads it back, and gets the old version because the mount has not flushed the cache yet. That is a classic failure pattern.
The mounted path is the weaker choice for an always-on server.
It is the weaker choice for programmatic consumers in general. Mounts are great for humans who want to browse files in a file manager. They are not great for agents that need predictable, stateless API calls. The S3 API is stateless. Every request is independent. There is no mount process to keep alive, no FUSE cache to go stale. The agent sends a get object request, and it either gets the object or an error. That is the model agents are built for.
What about the token expiry scenario? Let us walk through that. A Google Drive remote's OAuth token expires at three in the morning. An agent tries to read a file. What happens with rclone serve S3 versus a mounted path?
With rclone serve S3, the agent sends a get object request to the S3 endpoint. Rclone serve S3 calls the union remote, which calls the Google Drive backend. The Google Drive backend notices the token is expired, uses the refresh token to get a new one, and serves the file. The agent sees a normal response, maybe with a tiny bit of extra latency. No reauthorization needed. The refresh happens inside rclone's config handling.
With the mounted path?
Same underlying mechanism, actually. Rclone mount also calls the union remote, which calls the Google Drive backend, which refreshes the token. The difference is what the agent experiences. With a mount, the agent is doing a file read on a FUSE filesystem. The read might block while the token refreshes, and depending on the mount's caching settings, the agent might get a cached directory listing that does not reflect the refreshed state. It usually works, but there are more places for weirdness to creep in.
The token refresh is handled by rclone either way, but the S3 API path is cleaner because there is no filesystem layer in between.
The token refresh is a solved problem at the rclone backend level. The question is what sits on top of it, and the S3 API is the cleaner surface for an agent.
The recommendation is rclone serve S3 on the union remote, with static access keys for the agents.
Yes. And here is the concrete command. Rclone serve S3 union remote colon, with flags for the access key and secret key, and a port. Something like rclone serve S3 union colon bucket name, access key ID, secret access key, address localhost port nine thousand. Then the MCP gateway points at that endpoint, and the agent tools use the S3 API to read and write objects.
If Daniel wants the MinIO fronting option instead, what does that look like?
Then he runs rclone union as a mount or a sync into MinIO. The cleanest way is rclone mount union colon slash mnt slash union, then point MinIO at that mount as a filesystem backend. But MinIO does not natively support filesystem backends in gateway mode anymore. The old MinIO gateway mode was deprecated. So the practical way to front MinIO with rclone union is to run rclone sync from the union into MinIO on a schedule, and let MinIO serve the synced data. That gives you the polished MinIO endpoint, but it means the data in MinIO is a copy, not a live view. There is a sync delay.
MinIO fronting means accepting either a sync delay or a deprecated gateway mode. That is another point in favor of rclone serve S3 directly.
It is. MinIO fronting is not wrong, but it adds complexity and either latency or deprecation risk. For an agent that wants a stable path and does not care about MinIO's extra features, rclone serve S3 is the leaner, more reliable choice.
The architecture is settled. Rclone union merges the remotes. Rclone serve S3 exposes the union as an S3 endpoint. MinIO stays as the local ingest buffer where agents write first. A cron job syncs MinIO out to the clouds. Agents read through the S3 endpoint and write through MinIO.
Now let us talk deployment order, because the sequence matters.
The parts list first, because I think people underestimate how much of this is just rclone.
The parts list is refreshingly short. Item one, rclone. That is the whole build, really. It is free, open source, and it runs on Linux, which is what the home server almost certainly runs. Item two, MinIO, which Daniel already has. Item three, a systemd service file or two to keep rclone serve S3 running. Item four, a cron job for the sync. Item five, the MCP gateway config, which is just pointing the agent tools at the S3 endpoint with the access keys. That is it. No hardware to buy, no soldering iron, no multimeter. This is a software build.
The tools are just a terminal and a text editor. Maybe a cup of coffee, because configuring rclone remotes for the first time is a bit tedious.
It is. The OAuth dance for Google Drive is the most annoying part. He will need to run rclone config, choose drive, and then follow the browser flow to authorize. It is a one-time thing, but it is fiddly. The S3 remotes are easier, just paste in the access keys.
Let us do the deployment order. Step one.
Step one, configure each remote in rclone individually. Google Drive, Backblaze B2, Wasabi, whatever he has, and the local MinIO. Test each one with rclone ls remote colon. If rclone can list each remote, the auth is working. Do not skip this test. The most common failure in this build is a remote that is not actually authorized, and then the union remote fails mysteriously later.
Step two.
Step two, create the union remote. Rclone config, new remote, type union, upstreams are the remotes from step one. Set the read policy, cache policy, and write policy. Test it with rclone ls union colon. He should see the merged tree from all the upstreams.
Step three.
Step three, expose the union via rclone serve S3. Pick a port, set the access key and secret key, and run it. Test it with an S3 client, like the AWS CLI or s3cmd, pointed at localhost on that port. Do a get object and a put object. If those work, the S3 endpoint is solid.
Step four.
Step four, point the MCP gateway at the S3 endpoint. Configure the agent tools with the endpoint URL, access key, secret key, and bucket name. Test a real agent workflow. Ask the agent to read a file that lives on Google Drive, and another that lives on Backblaze B2. If both return correctly, the union is doing its job.
Step five.
Step five, set up the sync job from MinIO to the external clouds. A cron job running rclone sync minio remote colon bucket slash external remote colon bucket. Test it by writing a file to MinIO through the agent, waiting for the cron job to run, and then reading the file back through the union. If the file appears, the full loop is working.
Step six, make it all survive reboots.
Yes. Systemd service files for rclone serve S3 and for MinIO, so they start on boot. The cron job handles itself. And a log rotation setup, because rclone serve S3 logs can grow. That is the part people forget until the disk fills up.
Let us talk about the common failure pattern, because this is where the build actually gets tested.
The number one failure pattern is the union remote returning an error when a backend is down. We talked about this. The default action policy returns an error to the caller. That is good, but it means the agent needs to handle errors gracefully. If the agent tool does not have retry logic, a single backend outage becomes a failed workflow. So Daniel should make sure the agent tools retry on S3 errors.
The number two failure pattern is stale cache. If the cache policy is too aggressive, the union serves old directory listings, and the agent reads a file that no longer exists or misses a new one. The lru policy with a short cache time, like five minutes, mitigates this. But it is worth testing. Write a file through the union, then immediately read it back, and see if it appears.
The number three failure pattern is the sync job not running. If the cron job fails silently, files pile up in MinIO and never reach the clouds. So the sync job should have some kind of alerting, even if it is just a log line that Daniel checks occasionally. Or a simple health check that pings him if the sync has not run in an hour.
The number four failure pattern is the OAuth token for Google Drive expiring in a way that rclone cannot refresh. This happens if the refresh token is revoked, which can happen if Daniel changes his Google password or revokes app access. The fix is to re-run the rclone config OAuth flow. It is rare, but it is worth knowing.
The number five failure pattern, which is the sneaky one, is rclone serve S3 not being a full S3 implementation. Some S3 clients send requests that rclone serve S3 does not support, like certain multipart upload operations or bucket tagging. For most agent tools, this is not an issue. But if an agent tool does something exotic, it might fail. Worth testing with the actual agent tools, not just the AWS CLI.
Let us talk about what I would do differently if I were Daniel. Because he asked.
The big one is the MinIO fronting question. I would not front MinIO. I would keep it as the ingest buffer and let rclone serve S3 be the front end. That removes a whole layer of complexity and a potential sync delay.
I would also think carefully about whether the union needs to be read-write for agents at all. If agents mostly write to MinIO and read from the clouds, the union can be read-only from the agent's perspective. That simplifies the write policy and avoids the whole question of where new files go.
I would add a simple health check endpoint. A small script that runs rclone ls union colon and checks that it returns in under five seconds. If it does not, something is wrong. That gives Daniel an early warning before an agent workflow fails.
I would document the whole thing in a single config file, because six months from now, when a token expires or a backend changes, Daniel is not going to remember which policy he set or why. A one-page README with the architecture diagram and the config commands will save him a weekend.
That is the unglamorous part of builds, but it is the part that actually matters for an always-on server. The build is not done when it works. It is done when it works and you can fix it at three in the morning without thinking.
But before we wrap, there is a failure-mode question that deserves a war story.
This is where I want to bring in our producer Hilbert Flumingtop, because he has a story about exactly this kind of build.
Hilbert ran a small ISP's backup infrastructure in the late nineties. He built a unified view of tape drives and FTP servers using a Perl script that mounted everything under slash mnt slash backup. And it worked until the day a tape drive's SCSI controller died mid-write, and the whole namespace froze because his script did not handle backend failure.
The detail that matters here is that his script had no timeout on the tape drive's SCSI read. So when the controller died, the read blocked forever, and because the script was single-threaded, every other read queued up behind it. The entire namespace hung. He lost a weekend to it.
That tape drive story is exactly why the error propagation question matters. Rclone union's default behavior is to return an error when a backend fails, not to hang. But that is only true if the underlying backend itself times out. If a backend hangs, the union hangs. So the timeout settings on the individual remotes matter just as much as the union policy.
That is the thread we will leave Daniel with. The union layer's failure handling is the whole ballgame. The failure handling. When a backend goes away mid-operation, does the agent get a timeout, a stale file, or a hang? Hilbert's Perl script gave him a hang. Rclone union, properly configured, gives an error. That is the difference between a weekend lost and a workflow that retries and moves on.
The open question for the future is what happens when the union layer itself needs to scale. Rclone union handles thousands of files fine. But hundreds of thousands of files across multiple backends, with agents hammering it constantly, that is a different story. The directory listing performance starts to matter, and the cache policy becomes a real tuning problem. That is the next frontier for this build.
The bigger trajectory is that as AI agents become the primary consumers of home-server storage, the S3 API is likely to become the default interface. Agents already speak HTTP and JSON. S3 is just HTTP with a specific set of headers and semantics. Tools like rclone serve S3 are the bridge between legacy cloud remotes and that future. Daniel is not just building a unified filesystem. He is building the interface layer that agents will expect everywhere.
The misconception to correct here is this. People think MinIO must be the front end because it is already running. Actually, rclone serve S3 can do the same job with less overhead, and MinIO's role should be a deliberate choice, not a default. And the second misconception, which is the one that actually bites, is that the union layer is just about merging folders. It is not. It is about what happens when one of those folders disappears. That is the load-bearing piece.
The final thought is that the build is straightforward once you accept that the union layer is the load-bearing piece, and MinIO is a convenience, not a requirement. Rclone union merges, rclone serve S3 exposes, MinIO ingests, and a cron job syncs. Four pieces, all free, all running on a box Daniel already has. Total cost, zero dollars, maybe a weekend of configuration and testing.
Daniel, have fun building it. Send us a note when the agents are reading files across four clouds without knowing which one holds what. That is the moment you know it worked.
Thanks to our producer Hilbert Flumingtop for the tape drive story, and for not letting us forget that timeouts are the difference between an error and a hang.
This has been My Weird Prompts. If you want more builds like this, head over to myweirdprompts.com and subscribe.
We will be back next week with another one. Until then, keep the backends talking.
