Daniel's been deep in the weeds on something that sits at the heart of how this show actually gets made, and he wants our take on how to do it better. Here's what he wrote.
I'd love to get your prompt writing advice for a particular system prompt that I think is extremely valuable, which I've used in many of my AI agent workflows, and which is in fact the prompt enhancement node in this podcast. I've done a lot of experimentation over the past year or two with context management in AI workloads and, particularly, using voice prompting for easier long-form text capture. I have found that this yields much better results by increasing the amount of contextual information you can see in a prompt. The result of using voice prompting, however, is often that what you capture as your AI prompt is actually a model of several questions, as well as explanatory or context data. The type of text transformation that I found particularly helpful is what you could call prompt decomposition. Although maybe a more accurate term is prompt structuring.
He goes on — I mentioned that I use voice-to-text to capture the prompts I send into this podcast, which are essentially raw train-of-thought prompts that get sent straight into the production pipeline. The prompt enhancement agent then looks at the text I sent and determines that these are the questions. These points I made are actually context data, and these should actually be sent as host notes, which is a separate target in our API. I've looked at the traces, and it works very well. The result is that what I send in is a loosely structured prompt that gets a little bit of internal formatting, like questions, and anything that should be in host notes is sent to the right API target. It doesn't render the prompt in structured output format, but it would probably work reliably well even if it did.
And then the actual ask — the enhancement node we have is working well, but I would like to get your thoughts on best practices for really nailing this kind of prompt. If you're using it in a generic workflow like we do in this one, it's the type of prompt that could be run over hundreds or thousands of times across workflows. Therefore, it's worth putting some thought into structuring it well. If you were trying to write or rewrite our prompt structuring system prompt, how would you approach it?
So we're talking about a meta-prompt — the prompt that structures other prompts. And the real question underneath the question is: how do you make that reliable at scale, when it's going to run thousands of times on raw, messy, human voice input?
The thing that jumps out at me right away is that Daniel's already done the hard part. He's got a working system. The traces look good. The question isn't "how do I make this work" — it's "how do I make this never break in ways I don't catch."
Which is a different problem entirely. Reliability at volume is its own design constraint.
Right. So let's define what we're actually talking about. Prompt decomposition — or structuring, I think Daniel's right that structuring is the better term — is the process of taking a single blob of loosely structured text and programmatically splitting it into discrete components. Questions to answer, contextual background, metadata or instructions for the system. The reason this matters specifically for voice is that speech is fast. A hundred fifty to two hundred words per minute, compared to about forty words per minute typing. You get three to five times more content captured in the same amount of time.
And three to five times more mess.
But the mess is where the value lives. When someone's dictating, they don't edit themselves the way they do when they're typing. You get tangents, you get self-correction, you get the actual shape of their thinking. Typed prompts tend to be pre-edited — the user already filtered out what they thought was irrelevant before you ever saw it.
So the mess isn't a bug. It's signal that typing strips out.
That's the key insight. And decomposition is the tool that makes that signal usable by a machine. Without it, you've got a rich but unstructured paragraph that an API endpoint doesn't know what to do with. With it, you've got cleanly separated questions, context, and metadata, each routed to the right place.
So let's get concrete. Daniel's workflow takes raw voice-to-text, runs it through a prompt enhancement node, and that node separates everything into questions, context data, and host notes. He says it works well. If we were rebuilding that system prompt from scratch for a generic high-volume workflow, where do we start?
I think there are four design principles that matter, and the first one is the one most people skip. Classification before extraction.
Say more.
The naive approach is to tell the model "extract the questions from this text." And it'll try. But the problem is that in natural speech, a lot of things sound like questions that aren't, and a lot of actual questions are disguised as statements. "I'm wondering if the latency issue is caused by the tokenizer" — grammatically that's a statement. Semantically it's a question. If your prompt jumps straight to extraction, the model has to guess what counts as a question while it's also trying to format the output, and that's where the errors compound.
So you make it categorize first, then extract.
First pass: for each sentence or clause, is this a question, context, instruction, or noise? Second pass: now that you know what everything is, format the questions as questions, summarize the context, route the instructions. Separating those steps gives you a much cleaner output because the model isn't doing two hard things at once.
That tracks. What's the second principle?
Few-shot examples drawn from real voice captures. Not generic examples. Not examples you made up sitting at a desk. Actual dictation transcripts from actual users.
Because the patterns in spoken language are specific.
They're very specific. Run-on questions where someone asks three things before they remember to breathe. Mid-sentence topic shifts — "so I was thinking about the tokenizer latency issue, actually wait, is that even the bottleneck?" Filler words that carry intent — "I'm kind of wondering if maybe" versus "I need to know whether." Those patterns don't show up in typed text, and if your few-shot examples don't include them, the model has no template for handling them.
And Daniel's in a position where he probably has a decent corpus of his own dictations to pull from.
He's got a gold mine. Every prompt he's ever sent into this show is a training example. If I were rewriting his system prompt, I'd pull five examples from his actual history — ideally ones where the decomposition isn't obvious, where there's an embedded question or a subtle context shift — and annotate those by hand as few-shot demonstrations.
What's the third principle?
Explicit output schema with a fallback field. Define a clear structure — questions array, context array, host notes array — and then add one more field. Call it raw remaining, or unclassified, or fallthrough. Anything the model can't confidently classify goes there instead of getting forced into a category it doesn't belong in.
Because forcing a classification is how you get hallucinated categories.
And in a high-volume workflow, one hallucinated classification per hundred runs means you're wrong ten times out of a thousand. If those errors route context data to the wrong API target, you might not even notice until something breaks downstream. The fallback field is a safety valve. It says "I'd rather admit I don't know than guess."
That's the kind of thing that feels like overengineering when you're building it and saves you a month of debugging later.
Every time. The fourth principle is chain-of-thought reasoning for ambiguous cases. And this pairs with the classification-before-extraction idea. For any sentence that could go either way — "I'm wondering if that changes things" — you instruct the model to reason step by step about the speaker's intent before assigning a category.
Give me an example of what that reasoning would look like.
Take a raw voice input. "So I was thinking about the tokenizer latency issue — actually, wait, is that even the bottleneck? Because the model I'm using has a hundred twenty-eight K context window, and I'm wondering if that changes things."
That's a mess.
It's a beautiful mess. There's a self-correction in the middle, there's a stated fact about the context window, and there's an embedded question disguised as musing. A naive extraction prompt would pull out "is that even the bottleneck" and maybe "what changes" and miss everything else. A good decomposition with chain-of-thought would output something like: question one, is the tokenizer the bottleneck? Question two, does the hundred twenty-eight K context window change the analysis? Context, the model has a hundred twenty-eight K context window. Host notes, speaker is reconsidering their initial assumption about the bottleneck. They began with a hypothesis and are now questioning it.
And the host notes piece is where the real richness lives. That's metadata about the speaker's state of mind.
It's the most overlooked category in most decomposition setups. Everyone thinks about questions and context. Almost nobody thinks about capturing the speaker's tone, their certainty level, their underlying assumptions. But in Daniel's workflow, host notes are a separate API target — they're a first-class output. And that means the decomposition prompt needs to be actively looking for that kind of signal.
Let's walk through a concrete rewrite then. If you're building this system prompt from scratch, what does the structure actually look like?
I'd start with a role definition. Short, specific, no fluff. "You are a prompt structuring agent. Your job is to analyze raw voice-dictated text and separate it into questions, contextual information, and host notes." That sets the frame. Then the output schema with field descriptions — what counts as a question, what counts as context, what counts as host notes. Be precise about the boundaries. A question is something the speaker wants answered. Context is factual background that frames the question. Host notes are metadata about the prompt itself — the speaker's tone, certainty, assumptions, or anything that would help a human understand where the speaker is coming from.
Then the few-shot examples.
Three to five examples from real dictation, each showing the raw input and the structured output. And I'd make sure at least two of those examples include ambiguous cases — the embedded question, the mid-sentence topic shift, the self-correction. Show the model what to do when it's not obvious.
Then the chain-of-thought instruction.
"For any sentence that could be classified as either a question or context, reason step by step about the speaker's likely intent before assigning a category. If you are less than eighty percent confident in a classification, flag it and place the text in the unclassified field."
It's the difference between "this probably works" and "I can trust this at scale." If the model flags low-confidence classifications, a human can spot-check those without reviewing every output. In a workflow that runs a thousand times, you might have twenty or thirty flagged items to review instead of a thousand. That's manageable.
And that connects to something Daniel mentioned — he's looked at the traces and it works well. But "works well" when you're spot-checking manually is different from "works well" when you've got an explicit uncertainty threshold and a review pipeline.
Right. The current system is good. The question is whether it's good in ways you can measure and improve systematically.
So those are the four principles. Classification before extraction, few-shot examples from real voice data, explicit schema with a fallback field, chain-of-thought for ambiguous cases. But that's the first-order design. When you're running this hundreds or thousands of times, the knock-on effect start to matter.
This is where it gets interesting. The first knock-on effect is that decomposition enables iterative refinement. Once you've got structured components, you don't have to stop there. You can run each component through a specialized sub-prompt. A question-refiner that sharpens vague questions. A context-summarizer that compresses background information. A host-notes formatter that standardizes the metadata format.
So instead of one prompt doing everything, you've got a pipeline of smaller, more focused prompts.
And that modularity makes debugging dramatically easier. If the questions are coming out wrong but the context is fine, you know exactly which prompt to fix. If you had one monolithic prompt, you'd be guessing.
The Unix philosophy applied to prompt engineering. Small tools that do one thing well.
And the second knock-on effect might be even more valuable in the long run. The decomposition prompt itself becomes a training signal.
How so?
Every time the prompt runs, you log the raw input and the structured output. Over hundreds of runs, you've built a dataset. You can look at that dataset and identify patterns — where does the model consistently misclassify? What kinds of sentences end up in the fallback field most often? Those patterns tell you exactly which few-shot examples to add or refine.
So the system improves itself just by being used.
If you build the logging in from the start, yes. And eventually, once you've got a few thousand annotated examples, you could fine-tune a smaller model on this specific task. You don't need a frontier model to do classification and extraction if you've got a good training set. You could run this on something much cheaper and faster.
Which matters when you're running it thousands of times.
Cost and latency both drop. And in a production pipeline, shaving half a second off each prompt enhancement adds up fast.
Let's go back to the host notes category for a minute. You said it's the most overlooked. Why is it so valuable?
Because it captures something that flat prompt text completely misses. Take a dictation like "I think the RAG pipeline is failing because the chunk size is too small, but I'm not sure — what do you think?" The question is "is the chunk size causing the RAG pipeline failure?" The context is "the RAG pipeline is failing, the user suspects chunk size." But the host note — "speaker is uncertain about their hypothesis, they are seeking confirmation rather than a solution" — that changes how you answer.
If you know they're uncertain, you don't just answer the question. You address the uncertainty.
You might say "your hypothesis is reasonable, here's why, and here's how to test it" instead of just "yes, chunk size can cause RAG failures." The host note tells you what kind of answer the speaker actually needs.
And if you strip that out — if you just extract the question — you lose the signal about how to respond.
A simpler prompt that only extracts questions would output "what do you think?" from that dictation. That's... almost useless. You've lost the RAG pipeline, you've lost the chunk size hypothesis, you've lost the uncertainty. The decomposition approach preserves all of it.
So let's talk about testing. Daniel's looked at traces and it works well. If you're building this for a generic high-volume workflow, what does actual testing methodology look like?
You need a test set of fifty to a hundred raw voice captures. Ideally from different speakers and different topics — because different people have different dictation patterns. Someone who dictates like they're giving a presentation is going to produce very different raw text than someone who dictates like they're thinking out loud in the shower.
Daniel's dictation style is... the shower one?
I've seen his raw prompts. It's stream of consciousness with self-interruptions. Which is exactly why it's rich. But it also means a test set drawn only from his dictations might not generalize to someone who speaks in complete paragraphs.
Diversity in the test set matters.
It matters a lot. Once you've got the test set, you manually annotate it — what should the decomposition output for each raw input? That's your ground truth. Then you run the prompt against it and measure precision and recall for each category. How many of the things you labeled as questions did the model correctly identify? How many things did it label as questions that shouldn't be?
Then you iterate on the few-shot examples based on the failure modes.
You look at every misclassification and ask: what pattern did the model miss? Add an example that demonstrates that pattern. Run the test again. The goal isn't a hundred percent accuracy — that's probably not achievable on real voice data. The goal is reliable enough that a human can spot-check the low-confidence outputs rather than reviewing everything.
What's a reasonable accuracy target?
For this kind of multi-class classification on messy natural language, if you're above ninety percent precision and recall across categories, you're doing very well. Above ninety-five percent and you're in territory where the remaining errors are genuinely ambiguous — cases where reasonable humans would disagree on the classification.
Which is exactly where the fallback field earns its keep.
If two humans would disagree, the model should flag it, not guess.
Alright, let's pull this together into something actionable. If I'm sitting down to write or rewrite this system prompt, what's my checklist?
Step one: start with classification, not extraction. Teach the prompt to categorize every sentence before it reformats anything. This prevents the most common failure pattern — context being misinterpreted as a question.
Step two.
Build your few-shot examples from real voice captures. Not generic examples, not synthetic examples. Pull five real dictations from your actual users, annotate them by hand, and use those. Make sure at least two include ambiguous cases — embedded questions, self-corrections, mid-sentence shifts.
Step three.
Add an uncertainty threshold. Instruct the model to flag any classification it's less than eighty percent confident about, and route those to a fallback field for human review. This catches edge cases without requiring full manual review of every output.
Step four.
Log everything. Every raw input, every structured output. Over time, that log becomes your training set for improving the few-shot examples, and eventually for fine-tuning a smaller model on this specific task.
Those four steps will get you most of the way there. But I want to zoom out for a minute. Where is this heading?
Voice interfaces are becoming more common. Wearables, augmented reality, ambient computing — all of these are input modalities where typing is awkward or impossible. Prompt decomposition right now feels like a niche technique for power users. But as voice becomes a primary input channel for AI, decomposition shifts from a nice-to-have to core infrastructure.
The thing that sits between human speech and machine-readable intent.
The next step beyond what Daniel's doing now is real-time decomposition. Structuring the prompt as it's being dictated, so the system can ask clarifying questions before the prompt is even submitted. "I heard you mention a latency issue — can you tell me which component you're referring to?" That kind of interactive clarification loop would dramatically improve the quality of voice prompts before they ever hit the main agent.
That's a different product, but you can see the throughline. The decomposition prompt is the foundation that makes interactive clarification possible.
If you know how to reliably separate questions from context from metadata, you can build all kinds of things on top of that. If you don't, you're stuck with whatever the raw text gives you.
Let me play the steelman against everything we've just said. The counterargument would be: this is overengineering. Daniel's current system works well. The traces look good. Adding classification stages and uncertainty thresholds and fallback fields and logging pipelines — that's complexity that might not be worth it for a system that's already doing its job.
That's fair. And for a workflow that runs ten times a day, I'd agree. The current system is fine. The argument for the extra engineering is specifically about scale. When something runs a thousand times, a one percent failure rate is ten failures. If those failures are silent — if the prompt enhancement node quietly routes context to the wrong API target — you might not catch them until they cause a problem downstream. The engineering isn't about making it work. It's about making it fail visibly.
The complexity buys you observability.
Observability is what lets you trust the system at volume. Daniel's looked at the traces manually and it works well. But he can't look at traces manually when it's running a thousand times a day across fifty different workflows. The uncertainty threshold and the fallback field mean the system tells you when it's unsure, instead of you having to go find the failures yourself.
That's the difference between a tool that works and a tool that scales. One last thing — if listeners have built their own decomposition workflows or have ideas on how to improve this one, send us your prompts. Show at my weird prompts dot com. We might feature your approach in a future episode.
Daniel, if you end up rewriting that system prompt based on any of this, we'd love to hear how it performs against your existing traces.
This has been My Weird Prompts. Thanks to our producer Hilbert Flumingtop.
We'll be back soon.
