Every web app you use is a puppet. The strings are HTTP requests, and they're dangling in plain sight in your browser's network tab. Daniel's been deep in agentic AI projects, building MCP tooling that lets AI agents reach through the browser and pull those strings directly. He wants to know how to find the undocumented APIs the frontend calls, how to reverse-engineer them from network traffic, and crucially, how to do it in a browser-agnostic way. Not just scraping — he's asking how to read the browser's mail and hand the best bits to an AI.
And the thing is, the mail's not even sealed. The network tab is just... sitting there. Open by default. Every JSON payload, every GraphQL mutation, every REST call your browser makes is logged and inspectable. These aren't hidden in a security sense. They're undocumented, sure — nobody wrote you a nice OpenAPI spec — but they're fully visible. The frontend calls them with your session cookies, and you can watch it happen.
So the question isn't whether you can see them. It's which ones are worth handing to an AI agent, and how to find them without marrying yourself to Chrome DevTools.
Right. And there's a taxonomy that helps here. Category one: public endpoints called with your existing session. These are the low-hanging fruit — you open the network tab, filter by XHR or Fetch, interact with the page, and the requests just stream in. You can right-click, copy as cURL, paste it into a terminal, and it replays perfectly. Headers, cookies, payload — everything preserved. That's your starting point.
Category two being the ones that fight back.
Category two is where the frontend is doing something clever. CSRF tokens embedded in the page, request signatures computed in JavaScript, endpoints that only work if you called three other endpoints first in the right order. Those you can't just copy as cURL and fire off. You have to trace them back through the JavaScript bundle and understand the state machine the frontend is walking through.
Walk me through the discovery process for category one. I've got DevTools open. What am I looking for?
First filter. In the network tab, there's a bar that lets you filter by request type — you want XHR and Fetch. That immediately strips out all the images, stylesheets, fonts, the noise. What's left is the conversation between the JavaScript running in your browser and the server. Now interact with the page. Click a button, type in a search box, load a new view. Every meaningful action generates one or more requests.
And you're watching for JSON.
You're watching for structured data. The response preview tab is your friend. If you see an endpoint returning clean JSON with field names that make sense — labels, IDs, timestamps, status enums — that's a lever. If it returns an HTML fragment or a blob of minified JavaScript, it's probably not something you want your AI agent calling directly.
Give me a concrete example.
Search autocomplete. Open any modern web app with a search bar, open the network tab, filter to XHR, and start typing. You'll see a request fire — usually a GET with a query parameter like q equals whatever you typed — and the response comes back as a JSON array of suggestions. Labels, maybe IDs, maybe thumbnail URLs. That endpoint is almost certainly unauthenticated beyond your session cookie, and you can call it with any query string you want. Your AI agent now has a search lever.
And that took thirty seconds to find.
That's the thing. The discovery process for the easy stuff is shockingly fast. The harder cases are where it gets interesting. Say you're looking at a dashboard app — project management, CRM, something like that. You find the endpoint that creates a new item. You copy as cURL, replay it, and... four hundred three forbidden. Even though you're authenticated.
The CSRF token.
The server is checking for a header, usually X-CSRF-Token, and rejecting requests that don't have it. So now you've got to find where the frontend gets that token. First place to look: the page source. Right-click, view page source, search for csrf. Often it's sitting in a meta tag — name equals csrf-token content equals some long string. Or it's embedded in the initial page state, like window dot underscore underscore INITIAL STATE underscore underscore dot csrfToken.
And if it's not there?
Then you search the JavaScript bundle. In DevTools, go to the Sources tab, hit Control-Shift-F or Command-Option-F to search across all files, and search for the endpoint path you found in the network tab. Find the fetch call, and then walk up. Look for where the headers are being assembled. Somewhere in that call stack, the token is being read from a cookie, or computed from a value in local storage, or extracted from a previous response.
This is where it stops being a thirty-second job.
It becomes a puzzle. But it's a solvable puzzle, and the pattern repeats everywhere. Most web apps aren't doing anything exotic. The CSRF token is in a tag. The session ID is in a cookie the browser sends automatically. The request signature, if there is one, is usually some hash of the request body plus a timestamp — and you can find the hashing function in the JavaScript if you're patient.
What about request sequencing? You mentioned endpoints that only work if you called something else first.
This is the state machine problem. Some workflows require a specific sequence. You POST to create a draft, the server returns a draft ID, then you PUT to update the draft with content, then you POST to submit it. If you try to jump straight to submit with a made-up draft ID, it fails. You have to replicate the flow.
So you're essentially reimplementing the frontend's state machine.
In miniature, yes. You don't need to replicate every state — just the happy path. What does the frontend do when a user clicks through this workflow successfully? Watch the network tab as you do it manually. Note the sequence of requests, the response fields that get passed to the next request, any headers that change. Then your agent tool calls them in the same order.
That feels fragile. If the frontend team adds a step, your agent breaks.
It is fragile. That's the tradeoff. You're building on an undocumented, unsupported interface. The frontend team doesn't know you exist and wouldn't care if they did. Their job is to ship features, not to maintain API stability for bots.
Which brings us to the question of which endpoints are actually worth wrapping. You said search is a lever. What's not a lever?
Analytics pings. Click tracking. Anything that fires on a timer — heartbeat requests, session keepalives. These are noise. The test I use: if I described what this endpoint does to a human colleague, would they recognize it as a complete, meaningful action? Search for items, create a task, get my assigned issues, download a report — these are levers. Log impression, update last seen timestamp — these are not.
And intermediate steps? The save-draft before submit?
Compose them. Don't expose save-draft and submit as two separate tools your AI agent has to reason about. Wrap them into a single higher-level tool — create and submit item — that handles the sequencing internally. The agent calls one function, your tooling does the two or three API calls in order, and the agent gets back a single result. Reduces cognitive load and failure modes.
So we've been anchored on Chrome DevTools this whole time. Daniel asked about browser-agnostic approaches. What changes if you're not using Chrome?
Surprisingly little, but the tools shift. Firefox's network monitor is functionally equivalent — same filtering, same copy-as-cURL, same response inspection. Safari's web inspector can do it too, though it's less feature-rich and honestly a bit clunky for this kind of work.
And if you want to step outside the browser entirely?
mitmproxy. It's an open-source interactive HTTPS proxy. You run it on your machine, point your browser — any browser — at it as a proxy, and it captures every HTTP request and response passing through. It doesn't care what browser you're using. It doesn't even care if it's a browser at all — you can point a mobile app, a CLI tool, anything that speaks HTTPS at mitmproxy and it'll show you the traffic.
How does that work with TLS? The browser is encrypting everything.
mitmproxy generates its own certificate and acts as a man-in-the-middle. You install its CA certificate in your system trust store, and from the browser's perspective, mitmproxy is the server. It decrypts the traffic, logs it, re-encrypts it with the real server's certificate, and forwards it. The browser sees a valid TLS connection. You see plaintext.
Which is also why you shouldn't do this on a network you don't control.
Right. This is a local development technique. You're intercepting your own traffic on your own machine. Don't install random CA certificates on a work laptop without understanding the implications.
So the workflow is: fire up mitmproxy, point your browser at it, interact with the web app, and watch the requests stream by. Filter for JSON responses the same way you would in DevTools.
And mitmproxy has a Python API. You can write scripts that filter traffic programmatically, extract specific endpoints, even modify requests and responses on the fly. If you're building agent tooling at scale, you could write a mitmproxy script that watches for JSON responses, logs the endpoint URLs and payload shapes, and builds you a catalog of potential levers automatically.
That's moving from manual discovery to automated reconnaissance.
Which is where this gets interesting at scale. Right now, most people building MCP tooling are doing this by hand. Open DevTools, poke around, find the endpoints, write the wrapper. It works, but it's a bottleneck. Every new web service you want your agent to interact with, you do the whole dance again.
And the arms race you mentioned?
It's already starting. As more developers build agent tooling that calls these undocumented endpoints, frontend teams notice. They see unusual traffic patterns — requests coming from outside the browser, hitting endpoints in sequences the frontend never uses, at volumes that don't match human behavior. And they respond. Rate limits. Request validation. Obfuscated payload formats. Eventually, they deprecate the undocumented endpoints and move the logic behind a proper API gateway.
Which is, in a weird way, a good outcome.
It is. The hidden API layer is an accident. It exists because it was faster to build that way, not because anyone decided it should be a public interface. The pressure from agent tooling might actually accelerate the move toward well-documented, stable APIs. If your undocumented endpoints keep getting hammered by bots, the path of least resistance eventually becomes publishing a real API and telling people to use that instead.
Let's talk about authentication. Most of these hidden APIs rely on session cookies. How do you handle that in an agent tool?
Three approaches, each with tradeoffs. Option A: piggyback on the user's browser session. You extract the cookies from the browser's cookie store — which is possible but messy and browser-specific — and inject them into your HTTP client. This works until the session expires, at which point your agent breaks and the user has to re-authenticate manually.
Option B?
Implement the login flow programmatically. Your agent tool POSTs to the login endpoint with the user's credentials, captures the session cookie from the response, and uses it for subsequent requests. This is more reliable but means you're handling credentials, which is a security responsibility you may not want.
And option C?
OAuth tokens, if the service supports them. You register an OAuth application, the user grants access, and you get a bearer token that doesn't depend on browser sessions at all. This is the cleanest approach, but it only works if the service has an OAuth flow — and many of the services where you'd be reverse-engineering hidden APIs don't, because if they had OAuth they'd probably have a documented API too.
So you're often stuck with option A or B, both of which are brittle.
Brittle and, depending on the service's terms of service, potentially in a grey area. The act of inspecting network traffic and calling endpoints your own browser calls isn't illegal. The Computer Fraud and Abuse Act in the US is about unauthorized access — but you're authorized. You have an account. You're using your own credentials. The legal question isn't about the inspection, it's about whether calling the API directly violates the terms you agreed to when you signed up.
And terms of service are contracts, not criminal law.
Right. Breaking a ToS is a contractual dispute, not a hacking offense. But companies can and do sue over it, and they can ban your account. The practical risk for most individual developers building agent tooling is getting their API access cut off, not getting arrested.
Which brings us back to fragility. You said the frontend team doesn't know you exist.
And when they change something — rename an endpoint, add a required header, change the response shape — your agent breaks silently. The request returns a four hundred or a five hundred, or worse, it returns a two hundred with a different payload structure and your agent starts hallucinating based on garbled data.
So how do you detect that?
You build validation into your tooling layer. Before you pass the API response to the AI agent, you validate the schema. If the response doesn't match the shape you expect — missing fields, different types, unexpected nulls — you catch it and return a structured error to the agent instead of garbage. This is the same pattern as API versioning, except you're the one doing the version detection after the fact.
And then you go back to the network tab and figure out what changed.
Or your mitmproxy script alerts you. You could build a canary — a lightweight script that replays your known endpoints once an hour and checks the response schemas. If something breaks, you get a notification before your users do.
That's starting to sound like a whole discipline. Reverse-engineering frontends as a service.
I mean, it kind of is. There are companies that do this — API scraping services that maintain connectors to hundreds of web apps by reverse-engineering their frontends and keeping up with the changes. The difference now is that the consumer of the API isn't a data pipeline, it's an AI agent that needs to reason about which endpoint to call and what to do with the response.
And the AI agent doesn't care about the undocumented part. It just sees a function it can call.
From the agent's perspective, it's been given a tool. The tool has a name, a description, some parameters. The agent decides when to use it. What happens inside the tool — the HTTP request, the cookie management, the schema validation — is invisible to the agent. That's the whole point of the MCP architecture Daniel's working with.
So the craft is in choosing which tools to build and making them reliable enough that the agent can depend on them.
And the craft is also in knowing when not to build a tool. Not every endpoint deserves to be wrapped. The goal isn't to expose the entire surface area of the web app to the AI. It's to expose the small set of levers that map to the tasks you actually want the agent to perform.
What's a lever you might miss on first pass?
The one that returns context. Everyone thinks about the action endpoints — create, update, delete. But the endpoints that make an agent effective are the ones that tell it what's going on. Get my assigned items. List recent activity. Show me the state of this thing. An agent that can only write and can't read is just a bot with a flamethrower.
So you're looking for the read endpoints first.
Read endpoints with good filtering. A get all items endpoint that returns ten thousand records is worse than useless — it'll blow out your context window. You want the one that accepts query parameters for status, assignee, date range. The one that returns paginated results. That's the lever that lets the agent triage.
Let's talk about a specific case. Say I'm building an MCP tool for a project management app. Where do I start?
Open the app, open the network tab, and load your main view — the dashboard or the issues list. The first request you'll see is probably a GET to something like slash api slash issues with query parameters for your current filters. That's your read lever. Note the response shape — what fields does each issue object have? ID, title, status, assignee, due date. Those are the fields your agent will reason about.
Then I click create issue.
Watch the network tab. You'll see a POST, probably to slash api slash issues, with a JSON body. Note the required fields. Some of them might be surprising — a project ID, a template ID, a position in a list. Those are the hidden requirements that aren't documented anywhere. If you'd just guessed the API, you'd miss them and get a four hundred.
And the CSRF token?
Check the create issue page source for a tag. If it's there, your tool needs to do a GET on the page first, extract the token, then include it in the POST header. Two requests where the user sees one click.
This is the kind of detail that separates a tool that works sometimes from a tool that works.
And it's the kind of detail you only learn by watching the actual traffic. Documentation won't tell you. The frontend developers might not even remember they added it — it's probably a framework default.
So we've got manual DevTools, we've got mitmproxy for browser-agnostic capture, we've got schema validation and canaries for reliability. What's the thing people get wrong most often?
They assume the API is stable. They find the endpoints, build the tool, it works for a week, and then they stop thinking about it. But the web app is a living system. The frontend team is deploying changes. The API surface shifts under you. The tool you built is a garden, not a monument — it needs maintenance.
And the second thing?
They expose too much. They wrap every endpoint they find and hand the agent a toolbox with fifty functions. The agent gets confused, calls the wrong ones, chains them in weird ways. Better to give the agent five well-chosen tools that map cleanly to user intentions than fifty tools that map to API internals.
Less is more, but the less has to be exactly the right less.
That's the art of it. And the only way to develop that instinct is to build a few tools, watch how the agent uses them, and iterate. You'll learn which endpoints the agent reaches for and which ones it ignores. Prune the ignored ones. Refine the ones it uses. The tool surface should evolve with the agent's behavior.
Before we move on, I want to circle back to something. You mentioned that the hidden API layer is an accident — it exists because it was faster to build that way. But it's also, in a sense, the real API. It's the one the product actually uses. The documented API, if it exists at all, is often a second-class citizen — slower, less capable, missing endpoints the frontend uses every day.
That's absolutely true. The internal API is where the product investment goes. It's what the frontend team builds against. It gets the new features first. The public API gets them six months later, if at all. From the perspective of someone building agent tooling, the hidden API isn't just convenient — it's often the only way to access the full capabilities of the service.
Which is why this technique isn't going away, even if it's fragile.
It's not going away because the incentives are misaligned. The company wants you to use their app through their interface, where they can show you ads, track your behavior, control the experience. You want programmatic access. As long as that tension exists, people will be reverse-engineering frontends.
The companies that embrace it — that publish real APIs and treat them as first-class products — will win the agent ecosystem.
They'll win the power users and the developers building on their platform. The ones that fight it will find themselves bypassed anyway, just with more brittle integrations and angrier developers.
Hilbert: You're both talking about this like it's new.
Go on.
Hilbert: Twenty eighteen. I was doing QA at a travel booking startup. My job was to click through the booking flow fifty times a day and verify the prices matched. After about two weeks I wrote a Python script that watched the network traffic, pulled out the API endpoints, and ran the whole test suite in four minutes.
What were you using to capture the traffic?
Hilbert: Charles Proxy at first. Then I switched to mitmproxy because it had a Python API and I could script the whole thing. The script would log in, search for flights, pick one, go through the booking flow, and compare every price on every screen against a CSV of expected values.
This was all undocumented endpoints?
Hilbert: Every single one. The company didn't have a public API. The mobile app and the website both talked to the same backend, and neither one was documented. I just watched what the browser did and taught the script to do the same thing.
How long did that script last before something broke?
Hilbert: It broke about once a month. The frontend team would rename an endpoint or add a required field and my tests would start failing. I'd spend an hour fixing it, update the script, and we'd be good for another month.
That's the maintenance tax Herman was talking about.
Hilbert: The thing is, when my script broke, it was a test suite. The failures were visible. I knew exactly what changed because the script told me which assertion failed. Your AI agent won't tell you. It'll just start failing at its task, and the user won't know why.
Unless you build that validation layer.
Hilbert: Which most people won't. They'll ship the tool, it'll work for a while, and then it'll break silently. That's what I mean about scale and stakes. My broken script meant I spent an hour debugging. A broken agent tool means someone's workflow stops working and they don't know who to blame.
What happened when the CTO found out about your script?
Hilbert: He called me into a meeting and told me I was circumventing security controls. I pointed out that every endpoint I was calling was public, authenticated by the test accounts' session cookies, and that I was doing exactly what the browser did, just faster. He didn't care. He said I was creating unauthorized access paths.
Were you?
Hilbert: I was accessing the same URLs with the same cookies as the browser. The only difference was the User-Agent header. I changed it to match Chrome and he couldn't tell the difference in the server logs anymore. He dropped it after that.
The distinction between authorized and unauthorized came down to a header string.
Hilbert: It came down to whether he could tell it wasn't a real browser. That's the whole security model for most of these APIs. If it looks like a browser request, it's allowed. If it doesn't, it's suspicious. There's no actual access control beyond the session cookie.
Which is exactly the point we've been making. These aren't hidden in any meaningful security sense. They're hidden from documentation, not from access.
Hilbert: The other thing I learned: the endpoints that matter most are the ones that return lists. Search results, booking options, price calendars. The action endpoints are easy to find and easy to call. The list endpoints are where the data lives, and they're the ones that change most often because the product team is always tweaking filters and sort orders.
That maps to what you were saying about read endpoints being the real levers.
The list endpoints are where pagination gets interesting. If you're building a tool for an AI agent, you need to handle pagination cleanly. The agent shouldn't have to reason about page tokens and offsets. Your tool should abstract that — call the endpoint, collect all pages, return the combined result. Or better, accept a limit parameter and stop when you hit it.
Hilbert: My script did that. It would paginate through every available flight for a route and compare all of them. Took about thirty seconds for a busy route. The manual testers were doing one search at a time and writing down the prices in a spreadsheet.
How many testers did they replace with your script?
Hilbert: They didn't replace anyone. They just stopped making the manual testers do the price comparison part and had them do other things. The script ran alongside the manual testing. I don't think they ever officially acknowledged it existed.
That's probably the most realistic outcome for most of these agent tools too. They'll run alongside the human workflows, quietly doing the repetitive parts, never quite becoming the official interface.
Hilbert: Until they break and someone notices.
The one thing I'd take from this: the craft isn't in finding the endpoints. The network tab gives you those for free. The craft is in choosing which endpoints matter, composing them into reliable tools, and building the monitoring that tells you when they break. Everything else is just HTTP.
The thing I'd add: start with the read endpoints. The ones that tell your agent what's happening. An agent that can see is useful. An agent that can only act is dangerous. Get the context right, and the actions follow naturally.
Which leaves us with an open question. As AI agents get better at discovering and calling these hidden APIs on their own — and they will — do we end up with every web app effectively having two frontends? One for humans, one for agents? Or does the hidden API layer eventually become the primary interface, with the visual frontend just one consumer among many?
I think the pressure goes the other way. The more agents hammer these undocumented endpoints, the more companies realize that undocumented isn't the same as private. Either they lock them down — which is hard and expensive — or they document them and make them stable. My bet is on documentation winning, because it's cheaper than fighting a war you can't win.
Thanks to Hilbert Flumingtop for producing, and for the reminder that none of this is as new as we think it is.
This has been My Weird Prompts. If you're building agent tooling and you've got reverse-engineering war stories — or better yet, techniques we didn't cover — send them in. Show at my weird prompts dot com. We'll be back soon.
