Daniel has been building with MCP since it was a scrappy little protocol nobody had heard of, and he's been watching the ChatGPT plugin library the way some people watch a tide gauge. Not because he wants to use every plugin. Because he thinks the directory is the clearest public signal of where agentic AI is actually going.
And what caught his eye specifically?
The scientific corner of it. He expected finance and analytics, everybody expects finance and analytics. What surprised him was how substantial the ecosystem around scientific computing, drug discovery, and protein structure modelling has become. Highly specialised disciplines, and yet there's a whole shelf of tools for them.
Right, and his actual question underneath that is the interesting one. Not which plugins exist. Which of them have stopped being demos.
He wants the distinction between integrations that make existing software a bit easier to use, and integrations that expand what a researcher can do. Measurable time savings, new discoveries, previously impractical experiments becoming feasible. And then the big one. Whether general-purpose assistants are quietly becoming the access layer for serious scientific infrastructure.
That last part is the one I keep chewing on, because there's a version of it that's obviously true and a version that's mostly marketing, and they look identical from the outside.
Then let's start with what the directory actually is, because the number problem shows up immediately.
It does, and it's worse than most people realise. So plugins now bundle capabilities into reusable workflows across both ChatGPT and Codex, and the two products share one universal plugin directory. Under the hood an app needs an MCP server that defines its tools, and then optionally a web component that renders in an iframe. ChatGPT implements the open MCP Apps UI standard, so a UI you build once runs on any host that supports it.
Which is the smart part of the design. Build the interface once, it travels.
It travels, and that's why the counts exploded. Agent Discoverability is tracking five thousand seven hundred and twenty-three ChatGPT plugins as of today. Up four hundred and sixty-nine in seven days.
In a week.
Two hundred and thirty-eight thousand callable tools across them, nine thousand and eighty-six visible brands. And here's the part that should make anyone cautious, only fifty-two percent of the documented tools have descriptions that are actually useful.
So half the directory is a name and a shrug.
Half the directory is a name and a shrug. But the counting problem is worse than that. The MCP App Store indexed three thousand seven hundred and fifty-six ChatGPT apps back in September. The community awesome list counted one thousand six hundred and twenty-four apps across forty-four categories in July.
Three trackers. Three numbers. Overlapping dates.
And none of them is wrong, exactly. They're counting different things. One counts apps, one counts plugins, one counts MCP servers that may or may not be surfaced as plugins. So when somebody says the ecosystem has five thousand entries, ask them what an entry is.
A directory that can't agree on its own size is a directory that's still being built.
I'd put it more kindly, but yes. And the interesting question isn't the total anyway. It's the shape.
So if we can't agree on how many plugins there are, what can we say about the ones that matter for science?
We can say they're not where you'd expect to find them. Start with the categories that are dense and completely unsurprising. Analytics and Data, Finance and Trading. FactSet, LSEG, MSCI, CB Insights, Aiera, Datarails. Hex, Deepnote, MotherDuck, ThoughtSpot, Omni, PostHog, Mixpanel, Amplitude.
Those are obvious because the pitch is legible in one sentence. Connect the assistant to the data, ask questions, get answers.
And nobody needs domain expertise to see why that works. If you're already paying for FactSet, having the assistant reach into it is a straight line.
The finance shelf is the easy sell. Everybody understands wanting faster answers about money.
Now here's the twist that I think Daniel half-suspected. The directory's own ranking logic does not reward scientific tools. Agent Discoverability scores plugins, and the top of the ChatGPT leaderboard is Ashby at ninety-six, Supabase at ninety-three, Miro at ninety, Shopify at eighty-eight, Stripe at eighty-five.
So the winners are commercial SaaS with tidy tool descriptions.
That's exactly what the score is measuring. Discoverability rewards clean schema, clear descriptions, obvious naming. Wolfram and Consensus don't crack the leaderboards.
Wolfram, which does symbolic computation. Consensus, which synthesises the literature.
Neither of them scores like a project management tool. Which tells you the ranking is a map of commercial integration, not scientific depth. If you browsed that leaderboard looking for the frontier of research tooling, you'd walk away thinking there isn't one.
That's the trap. The surface of the directory is optimised for a different question than the one Daniel's asking.
And it's worth saying out loud what the visible scientific entries actually are. Wolfram appears as add computation and knowledge. Symbolic computation, curated knowledge, formal math inside chat, from the team behind Wolfram Alpha and Mathematica. Consensus retrieves papers from an indexed academic corpus, generates a synthesis, and then shows you a Consensus Meter, which is a little readout of how much the retrieved papers agree with each other.
That meter is the interesting design choice. It's not just giving you a paper list, it's giving you a sense of the state of the disagreement.
Because the disagreement is the finding, half the time. But here's the thing, and this is what I want to get to. Neither of those two is a drug discovery tool. Wolfram is general computation, Consensus is literature search. The capability-expanding work in drug discovery is almost entirely absent from the plugin directory.
So where is it?
In the arXiv agent literature. Which is a completely different ecosystem, with different incentives, and it hasn't migrated into the directory yet. LIDDIA, revised in October, generates molecules meeting key pharmaceutical criteria on over seventy percent of thirty clinically relevant targets. It identified a novel candidate on a receptor called AR, NR3C4.
Seventy percent on thirty targets. That's not a demo number.
It's a real number, and so is MADDIE, which is a multi-agent engine that was tested across fifty protein targets. DrugPilot reports task completion of ninety-eight percent on simple tasks, ninety-three and a half on multi-tool, and sixty-four on multi-turn scenarios.
That last number is the honest one.
That last number is the honest one, and I appreciate that they published it. Everything degrades when you chain it. There's a survey from October of last year, forty-five pages, that claims to be the first comprehensive work presenting real-world implementations and quantifiable impacts of agentic systems in operational drug discovery. It reports workflows that once took months now taking hours.
Now taking hours, with what in the loop?
That's the question the survey doesn't fully answer, and we'll come back to it. But the mechanism part is real. AlphaFold 3, released back in May of twenty twenty-four by DeepMind and Isomorphic Labs, predicts joint structures of proteins, nucleic acids, small molecules, ions, modified residues. That's the foundation everybody builds on.
And the building on top is where it gets interesting.
It is. There's Bayesian refinement of AlphaFold models for in-silico drug design, published in July. There's DrugCLIP, which does genome-wide virtual screening using contrastive learning on AlphaFold-predicted structures.
So AlphaFold gave you the structure, and now the field is trying to turn structures into decisions.
That's the whole second act. And it's contested. There's a preprint from July that evaluates AlphaFold 3 across diverse application areas and documents specificity gaps and training-data dependence. The framing is basically, how well do the new capabilities actually translate into application tasks.
And the answer isn't a clean yes.
The answer is, sometimes, and less often than the reviews suggest. That's the pattern I keep seeing. The capability is real, the performance on an actual task is variable.
So here's what I'm hearing. The scientific ecosystem is real and it's growing. But it's split in two. A thin visible layer of general-purpose tools in the directory, Wolfram and Consensus, and then a much deeper layer of specialised agent research happening in papers that haven't become plugins at all.
That's the split, and I'd go further. The plugin library is not yet the map of scientific AI. It's the map of commercial AI with a few scientific postcards pinned to the corner.
A postcard is cheaper than a lab.
Right, and that's why the counts look healthy while the thing Daniel actually cares about is still mostly in research code. Which takes us to the impact question, because that's where the evidence cuts both ways.
Which impact question?
Whether any of this is measurably reducing research time, or producing discoveries, or making impossible experiments possible. And the honest answer is that the closest measurable evidence isn't plugin evidence at all. It's model-level and agent-benchmark-level.
Give me the strongest version of the positive case.
LEADS. A medical literature foundation model, published in Nature Communications. Trained on six hundred and thirty-three thousand samples drawn from twenty-one thousand systematic reviews.
That's a lot of reviews.
It's the whole visible literature of evidence synthesis. And then they did a user study with sixteen clinicians and researchers from fourteen institutions, which is small but it's real humans doing real work. Experts using LEADS hit zero point eight one recall in study selection, versus zero point seven eight working alone. Twenty point eight percent time savings. In data extraction they went from zero point eight zero to zero point eight five accuracy, with twenty-six point nine percent time savings.
So roughly a quarter of the grunt work, gone.
Roughly a quarter of the grunt work, gone, with humans still in the loop doing the judging. That's the shape of the win.
Now the countervailing case. Because I know there is one.
There is one, and it's ugly. A systematic review in Research Synthesis Methods looked at GenAI for evidence synthesis and found it missed sixty-eight to ninety-six percent of studies during searching. Median, ninety-one percent.
It missed nine out of ten studies.
Nine out of ten. And the conclusion is blunt. The current evidence does not support GenAI use in evidence synthesis without human involvement or oversight. Then a separate review on data extraction found accuracy ranging from forty-seven percent to ninety-nine point nine, with numerical data much worse, forty-seven to eighty-eight percent, and categorical data much better, seventy-four to ninety-six percent.
So it can read the paper but not the table.
It can read the paper and it will confidently misread the table. Numbers are where it falls apart, and numbers are the entire point in half of these workflows.
And that gets us back to the distinction Daniel drew. Integrations that make existing software easier to use versus integrations that expand what you can do. Wolfram in chat is the first kind.
Wolfram is a real accessibility win. Natural language into symbolic computation means a biologist who never learned Mathematica syntax can get a symbolic answer. That's useful and it is not a new capability. The math was always there.
And LIDDIA, MADDIE, those are the second kind. They generate molecules that didn't exist.
They propose candidates, which is a capability, not an interface. But they're not in the plugin directory. They're in research code and papers, and the gap between a paper and a tool a lab can rely on is enormous.
What's in that gap?
Three things, if you believe the deployment paper from March. It cites over ten thousand active MCP servers and ninety-seven million monthly SDK downloads as of early this year, which is enormous. And then it names three missing production primitives. Identity propagation. Adaptive tool budgeting. Structured error semantics.
Say what those mean in practice.
Identity propagation means when the agent calls a tool on your behalf, the tool knows it's you, with your permissions. Not the agent, not a generic service account, you. Today that's mush. Adaptive tool budgeting means the agent knows how much of each tool it can afford to call before it stops being worth it. Structured error semantics means when a tool fails, it fails in a way the model can actually recover from, instead of returning a sentence that says something went wrong.
So the demo works and the production thing doesn't, because the demo never had to handle a failure at three in the morning.
And there's a security tax on top of it that nobody wants to talk about. A security analysis from January found MCP's architecture amplifies attack success rates by twenty-three to forty-one percent compared to equivalent non-MCP integrations.
Compared to the same integration without MCP.
Same functionality, different plumbing, more successful attacks. And their proposed extension, MCPSec, cuts attack success from fifty-two point eight percent down to twelve point four.
So the fix works and it isn't standard yet.
The fix works in the paper. Whether it's in the thing you installed is a different question. And this matters more for scientific tools, because a literature search that gets poisoned is annoying. An instrument controller or a clinical dataset that gets poisoned is a different category of problem.
Which is interesting, because Daniel framed the whole thing as an access layer. General-purpose assistants becoming the front door to sophisticated infrastructure.
And that's happening. It's just that the front door currently opens onto a lobby with a lot of commercial SaaS in it and a small research annex off to the side. The access layer is real. The sophistication behind it is thinner than the directory's size suggests.
So augmentation works and autonomy doesn't. That's the one-line version.
With an asterisk, which is that we don't have plugin-level evidence for either. LEADS is a model. LIDDIA is an agent benchmark. Nobody has published the study that says a researcher using Consensus inside ChatGPT saved eleven hours this month. The evidence is one layer down.
Does that bother you?
It bothers me professionally. The mechanism is the same, so I'd expect the direction to hold. But I've been burned before by expecting a result to survive one layer of indirection, and I'll say so rather than pretend the study exists.
And the failure modes here are weirdly specific.
The word you keep using is wrong.
Which word?
Plugin. You keep saying plugin like it's a thing. My niece works in a lab, and she'd say you're describing a door.
Go on.
She rang me two weeks ago. Her lab tried one of those AI research assistants for a literature review. It cited a paper. Title, authors, journal, DOI. The DOI resolved to nothing. The paper does not exist. It was never published. It was never written.
That's a hallucinated citation, which we've all seen.
It's worse than a hallucinated citation. The title was specific. It was on topic, it had the right words in it, and the postdoc spent a full day trying to find it through interlibrary loan before he gave up.
A full day on a paper that doesn't exist.
Two days across the lab, if you count the second person who looked. And that's the part you can't put in a benchmark. It isn't that the tool lied. It's that the lie was good enough to be chased.
That is the weak spot I'd worry about more than the obvious one. An obviously wrong answer costs you nothing. A plausible wrong answer costs you a day.
My niece called me specifically to ask if I'd heard of the journal.
And had you?
I told her I'd look into it. I was in publishing for a while, so I know how to check these things.
Were you in publishing?
Circulation. I was a circulation consultant for a regional magazine. It folded. Not because of the circulation. But I know what a real journal looks like. I have opinions about impact factors.
Of course you do.
She checks every citation by hand now. All of them. She says it's slower. I told her that's probably fine.
Is it fine?
It's what she's doing, and nothing has gotten past her since.
The tools are getting better at sounding right.
That's the thing. Sounding right is exactly the property that makes the failure expensive, because a researcher's whole workflow is built on trusting the citation.
And the fix is a human checking each one by hand. Which is the least futuristic possible solution to a problem created by the most futuristic possible tool.
I don't think there's anything wrong with that. You check the thing you're going to build on. That's the whole job.
Nobody puts that on the box though. The box says synthesise the literature. It doesn't say synthesise the literature and then verify every reference manually before you trust a sentence of it.
Which is the honest picture. Augmentation, not autonomy, and the human in the loop is doing the least glamorous part.
That's what Daniel asked directly, whether there are measurable reductions in research time or previously impractical experiments becoming feasible. The answer is that the reductions are real when a human is judging and fictional when nobody is.
Let's do the correction, because I think there's one belief that needs killing.
There is, and it's the one most people hold without noticing they hold it. The belief is that the size of the plugin library measures scientific capability. Five thousand seven hundred and twenty-three plugins, two hundred and thirty-eight thousand tools, so science must be deep in there somewhere.
It doesn't. The leaders on discoverability are Ashby, Supabase, Miro, Shopify, Stripe. Wolfram and Consensus aren't up there. The directory's surface tells you about commercial integration. The scientific depth is somewhere else, mostly in research code.
The second belief underneath it, that these tools are either working or not working, is also wrong. The evidence splits cleanly. With human oversight you get measurable time savings, a fifth to a quarter of the work in some tasks. Without it you get a median ninety-one percent miss rate. It's not one verdict. It's two, depending on whether a person is in the loop.
The honest place to land is that the access layer is real, the science is real, and the gap between demo and tool is still the story. Which raises the open question, do the scientific tools migrate into the plugin layer, or do they stay in research code and specialised platforms?
It depends on whether identity propagation, adaptive tool budgeting and structured error semantics get solved, and whether the security tax gets paid down. Those three primitives are what separate a demo from something a lab can build on. Until they exist, the serious tools stay where the serious engineers are, which is not the plugin directory.
The shape of the transition might be the finding. The researchers getting value are the ones staying in the loop. Maybe that isn't a limitation waiting to be overcome. Maybe that's what this actually looks like for a while.
A tool that needs a human checking its citations is still a tool that saves you twenty-five percent of your week. That's not nothing. It's just not the story being sold.
Thanks to Hilbert Flumingtop, who produces this show and checks the things he's going to build on.
If you want more of this, try episode fourteen, AGI's Crossroads; episode twelve, The AI Breakthrough; and episode ten, How ASR Went From Frustration To ... Whisper Magic. This has been My Weird Prompts.
If you want to send us a prompt of your own, the line is send us your own prompt on Telegram at t dot me slash MWP listener bot.
We'll be back soon.
See you tomorrow.
