Thirty percent. That's how much of one of the largest public training corpora was duplicate or near-duplicate content before anyone thought to clean it.
And that's a corpus people were building language models on. The podcast pipeline has a much smaller problem, but the same failure shape: something fires twice, nobody notices until the feed has two episodes with the same title and Daniel has to go prune it by hand.
Daniel's prompt this week is about exactly that. He's walking through what happens when he submits a voice prompt from the road, the connection drops before the form gets its success message back, and the form sits there looking like it failed. So he resubmits. But the webhook actually reached the pipeline the first time, the generation is already running, and now there are two of them.
The rebound traffic dropped. That's the phrase he used. The request made it, the response didn't.
Right. So he's asking how to build a duplicate screener that catches this before it burns API credits. And he's drawing a distinction between two paths. The audio upload path is clean: checksum the file, store the checksum, compare on the next submission. The text path is messier, because he might change a character or add paragraph spacing between attempts, and an exact string match would miss it.
And he wants the threshold question handled too. Not just exact match, but minimal material change. That's the actual engineering problem here.
So let's do the audio path first, because it's the one that shows the pattern without the ambiguity. Then we can get into why text matching is where people quietly get it wrong.
The audio checksum is a SHA-256 hash of the file bytes. Deterministic, collision-resistant for practical purposes, and sensitive to a single byte change. If Daniel uploads the same audio file twice, the hashes are identical. If he re-encodes it, even at the same bitrate, the hash changes because the bytes changed.
So the checksum is a fingerprint, but only for the exact file.
Only for the exact file. And for this use case, that's fine. Daniel's failure mode is the same file being submitted again because the form didn't clear. He's not worried about someone re-encoding the audio and accidentally creating a near-duplicate. He's worried about the same bytes going through the pipeline twice.
The implementation is a lookup table. Store the checksum with the submission, and before anything else runs, check whether that checksum already exists. If it does, kill the submission right there.
The important detail is where the check lives. It has to be before the generation API call. If the duplicate check runs after generation, you've already spent the credits. The whole point is to catch it at the webhook, before the expensive work starts.
Daniel said that explicitly. The duplicate screener kills the pipeline before it gets any further. That's the right instinct.
And the database constraint is what makes it actually work under concurrency. If two submissions with the same checksum arrive at nearly the same time, a check-then-insert pattern races. Both workers check, neither sees the checksum, both proceed. The fix is a uniqueness constraint on the checksum column, and an insert that does nothing on conflict. The claim and the write become one atomic operation.
So the database is the bouncer. Not the application logic.
The database is the only place that can enforce processed-at-most-once under concurrency. Application-level checks are advisory. The constraint is the guarantee.
That's the audio path. One hash, one table, one constraint, done. Now the text path.
The text path is where Daniel's instinct to reach for a string match runs into the normalization problem. He said it himself: if he removes a trailing character, or adds paragraph spacing, the exact string changes. So a naive equality check fails.
But the fix for a lot of that isn't fuzzy matching. It's normalization before hashing.
Right. The thing people miss is that raw text encodes the same content with different whitespace, different Unicode forms, different line endings. If you hash without normalizing, a spacing-only edit produces a different hash and you keep both copies. The dedupe pipeline thinks it's working and it's silently keeping near-identical duplicates.
So the first layer is cheap. Lowercase the text, strip leading and trailing whitespace per line, collapse consecutive whitespace to a single space, normalize Unicode to NFC. Then hash.
And that kills the paragraph spacing case immediately. If Daniel adds three blank lines between paragraphs, the normalized text is identical to the original. The hash matches. The duplicate screener catches it.
The trailing character case is different. If he deletes the last sentence because he thinks it didn't send, that's a material change to the content. Normalization won't catch that. You need similarity scoring.
That's where the threshold question Daniel raised actually lives. He's asking: did this string have no or minimal material changes? That's not a boolean. That's a score.
So what's the scoring approach?
The standard tool is Jaccard similarity over character shingles. You take the text, break it into overlapping chunks of five to eight characters, and compare the set of chunks between two documents. The similarity is the size of the intersection divided by the size of the union. Two texts that are identical except for a deleted trailing character score very high. Two completely different prompts score near zero.
And then you pick a threshold above which you call it a duplicate.
That's the tuning problem. The threshold determines how aggressive the dedupe is. There's a worked example with a hundred fifty documents, fifty unique and a hundred noisy copies. At a threshold of zero point one, the system removed ninety-two percent of the corpus. That's over-aggressive, it's collapsing things that aren't really duplicates. At zero point four, it removed less than one percent. The sweet spot in that example was around zero point two to zero point three.
None. For Daniel's use case, the threshold depends on how much change he wants to tolerate before two submissions count as different prompts. If he sometimes tweaks a sentence and resubmits on purpose, a high threshold lets that through. If he wants to catch any near-resubmission, a lower threshold catches more but risks blocking a legitimate second prompt that happens to share phrasing.
And his prompts probably share a lot of phrasing across different episodes. He asks about pipelines, about automation, about AI workflows. There's a real risk of false positives if the threshold is too loose.
That's the thing to watch. Character shingles are language-agnostic and tolerant of small edits, which is good. But two different prompts about similar topics will share some shingles. The threshold has to be calibrated against his actual prompt corpus, not borrowed from a document dedupe benchmark.
There's a simpler tool for the fuzzy case, though. RapidFuzz.
RapidFuzz is the practical answer for a pipeline like this. It gives you a normalized similarity score from zero to a hundred, and it has different comparison modes. The default ratio is fine for small edits. Token sort ratio neutralizes word reordering. Token set ratio handles cases where one string has extra tokens. And each one accepts a score cutoff so you only get matches above your threshold.
For address-style strings, the guidance is a threshold around eighty-eight to ninety. For prompts, I'd want to see what his actual near-duplicates score before picking a number.
Agreed. The calibration step is: take the last fifty prompts, compute pairwise similarity, and look at the distribution. Find the gap between the scores of genuine near-duplicates and the scores of different prompts. Put the threshold in that gap.
That's the part nobody does. They pick a threshold from a blog post and ship it.
And then they're surprised when the screener either misses duplicates or blocks legitimate submissions. The threshold is a property of your data, not a universal constant.
There's a deeper design issue here, though. Daniel's instinct is to hash the prompt content and block repeats. That's content dedupe. But there's a standard warning in webhook engineering against using a hash of the body as your idempotency key.
Because two legitimately distinct events can share a body. Two identical refunds a minute apart. Two identical prompts sent on purpose because Daniel wants to regenerate an episode with a different model. A content hash would collapse those into one.
So there are two different things going on. Transport idempotency, which is about making sure the same submission attempt doesn't get processed twice. And content dedupe, which is about making sure the same prompt doesn't generate twice. They're related but they're not the same.
The robust pattern is two layers. The transport layer uses an idempotency key that's unique per submission attempt. The client generates it, or the server derives it from something that identifies the event, not the content. That key goes in a processed events table with a uniqueness constraint. Same event arrives twice, the second one is dropped before any work happens.
And the content layer is the checksum or the similarity score, which answers a different question: has this prompt, or something materially identical to it, been run before?
The content layer is what catches the resubmission after a dropped response. The transport layer is what catches the webhook retry. Both are worth having.
Daniel's failure pattern is actually the transport problem manifesting as a content problem. The webhook fired, the response dropped, the client thought it failed. If the client had sent an idempotency key with the original submission, the resubmission would carry the same key and the server would drop it immediately.
That's the cleanest fix, honestly. The form generates a unique submission ID when Daniel starts filling it out. That ID goes with the request. If the response drops and he resubmits, the same ID goes with the second request. The server sees the ID in the processed table and says no.
But that requires the form to preserve the ID across the resubmission. If the form clears or reloads, the ID might regenerate.
It depends on how the form is built. If the ID is generated on page load and kept in a hidden field, a resubmission from the same page keeps it. If Daniel closes the tab and reopens the form, the ID is new and the transport layer can't help. That's where the content layer becomes the backstop.
So the two layers cover different failure pattern. Transport catches the retry from the same page. Content catches the resubmission from a fresh form.
And the content layer is the one that needs the fuzzy matching, because a fresh form means Daniel might have retyped or edited the prompt.
There's an economic argument for where the gate lives, too. Daniel mentioned API credits. If the duplicate check runs before the generation call, a duplicate costs almost nothing. A hash lookup, maybe a similarity computation. If the check runs after generation, the duplicate costs a full generation's worth of credits plus the manual pruning time.
The claim-first pattern is what makes the early kill possible. When a submission arrives, you claim it in the database with a status of processing. Then you do the expensive work. Then you mark it done. If a duplicate arrives while the first one is still processing, the uniqueness constraint blocks it before the second generation starts.
And if the first one crashes mid-processing, you need a reclaim rule. A row stuck in processing after some timeout gets reclaimed and retried.
That's the crash recovery story. Without it, a crashed worker leaves a row in processing forever and the prompt never runs. The timeout depends on how long generations take. If a typical generation is two minutes, a reclaim after fifteen minutes is probably safe.
This is all very doable for the pipeline Daniel's running. The pieces are all off-the-shelf.
The audio path is maybe twenty lines of code. The text path is normalization plus a similarity threshold, which is a bit more work but still small. The idempotency layer is a table and a constraint.
The hard part isn't the code. It's the calibration. Picking the threshold, testing it against real prompts, adjusting when it misfires.
And accepting that no threshold is perfect. There will be false positives and false negatives. The question is which failure is cheaper. For a podcast pipeline, a false negative means a duplicate episode slips through and Daniel prunes it manually. A false positive means a legitimate prompt gets blocked and Daniel wonders why his episode never generated.
The false positive is worse, in my view. A duplicate episode is embarrassing but recoverable. A silently blocked prompt is invisible. He might not notice for days.
That argues for a conservative threshold. Err toward letting things through, and treat the screener as a backstop rather than a gatekeeper.
Or add an override. If the content hash matches but Daniel wants to rerun the prompt, let him force it through.
The hybrid approach. Content hash plus explicit override. The screener flags, the human decides. That's the right amount of automation for a pipeline where the operator is also the content creator.
And for the audio path, the override is almost never needed. If the same audio file is uploaded twice, it's a duplicate. No ambiguity.
Unless he re-encodes the audio and uploads a different file with the same content. Then the checksum misses it. But that's not his failure pattern, and chasing perceptual audio hashing for that case is over-engineering.
Perceptual hashing is for when you care about content identity across formats. Shazam-style landmark hashing, chromaprint, that sort of thing. It's what you'd use if you were deduping a music library with mixed encodings.
For a single uploader using a single form, the plain checksum is the right tool. Daniel said it himself: the audio path is the cleaner teaching point.
The text path is where the real engineering lives, because text is squishy. Whitespace, Unicode, edits, reordering. The same prompt can be encoded a dozen ways and still be the same prompt.
And the normalization layer is the unglamorous hero. Collapsing whitespace and normalizing Unicode kills most of the false-negative cases before you even reach for a similarity score.
I keep thinking about that thirty percent figure from the corpus work. If a training corpus has thirty percent duplicate content, and the people building it didn't notice, then a small pipeline with one human operator is going to have the same blind spot.
The difference is scale. At corpus scale, you need MinHash and suffix arrays and Bloom filters. At Daniel's scale, a checksum column and a similarity threshold are plenty.
There's a principle there. Match the tool to the scale. Don't build a distributed dedupe system for a single-user pipeline.
The other principle is: persist before responding. Never process inline. Write the payload to the database, return a fast success response, and let an async worker do the generation. That's the pattern that prevents the dropped-response problem from causing duplicates in the first place.
Because if the server responds fast, the form gets its success message before the connection drops. The failure window shrinks.
The webhook was reached, the pipeline was running, but the response was slow enough that the rebound traffic dropped. If the server had persisted the payload and returned immediately, the response would have made it back before the connection died.
So part of the fix isn't dedupe at all. It's making the acknowledgment fast.
Persist before responding. Never process inline. Idempotency key required. That's the practitioner consensus for webhook handling.
And the idempotency key is the piece that makes the retry safe. If the provider retries because it didn't get the response, the server sees the same key and doesn't process twice.
Daniel's form is the provider in this case. The form sends the webhook. If the form doesn't get the success message, it retries. If the retry carries the same submission ID, the server drops it.
So the full architecture is: fast acknowledgment, transport idempotency key, content checksum for audio, normalized hash plus similarity threshold for text, database uniqueness constraints, and a human override for edge cases.
That's a complete answer. And none of it is exotic. It's all standard database and hashing work.
The thing I'd tell Daniel to build first is the audio checksum table. It's the simplest piece, it solves the cleanest case, and it establishes the pattern for everything else.
Then the text normalization layer, because that's also cheap and it catches the spacing-only edits. Then the similarity scoring with a conservative threshold, calibrated against his actual prompt history.
The idempotency key on the form, which is the piece that prevents the retry from ever reaching the content layer.
The form change might be the highest-leverage fix, actually. If the form generates a submission ID and keeps it across resubmissions, most duplicates die at the transport layer before any hashing happens.
But only for resubmissions from the same page. Fresh form, new ID, content layer has to catch it.
Right. Defense in depth. Each layer catches what the layer above it misses.
Daniel's going to have to tune the text threshold himself, though. We can give him the tools and the approach, but the number comes from his data.
He should log every time the screener fires, so he can review false positives and adjust. The threshold isn't set once. It's maintained.
A screener that silently blocks prompts is worse than no screener. Visibility matters.
Log the blocked submission, store the similarity score, let Daniel see why it was blocked. If it was a false positive, he can override and the system learns.
Or he just adjusts the threshold manually. No need for a learning system. A log and a config value are enough.
Hilbert: The Panasonic KX-TG two thousand four hundred.
...What?
Hilbert: Cordless phone, late nineties. Had a redial button that would resend the last number you dialed. If the line was flaky, you'd hit redial and it would send the whole thing again. Same number, same call. The phone company didn't dedupe anything. You'd get two calls to the same person and they'd pick up the second time and say, I just talked to you.
The transport layer there was the phone network, and it had no memory at all.
Hilbert: It had plenty of memory. It remembered the last number. It just didn't know the difference between a new call and a retry. Same problem Daniel has. The form remembers the prompt, sends it again, and the server doesn't know it's a retry.
The fix is to give the retry an identity.
Hilbert: That's what the ID is for. We used to write down the time we called so we'd know if the second call was the same one. Low tech, same idea.
The submission ID is the digital version of writing down the time. It travels with the request and the server can recognize it on the second arrival.
Hilbert: The checksum on the audio is cleaner, though. A phone call didn't have a checksum. You couldn't hash a conversation. But a file, you can hash it and know for certain it's the same file. That's better than anything we had.
The audio path is the one place where the answer is unambiguous. Same bytes, same hash, done.
Hilbert: I spent a summer digitizing cassette tapes for a radio station. We'd record the same tape twice sometimes because the first transfer had a dropout. The second transfer was a different file, different bytes, but the same audio. A checksum wouldn't have caught it. We had to listen and compare.
That's the perceptual hashing case. Same content, different encoding. For Daniel's audio uploads, the file is the same file, so the checksum works.
Hilbert: Right. He's not re-encoding. He's resending. Different problem.
The text is the part I keep coming back to. The threshold question. How much change is too much change before it's a new prompt?
Hilbert: When I typed up the station logs, if I added a comma, it was still the same log entry. If I changed the call letters, it was a different station. Somewhere in between was a judgment call.
That's the material change question Daniel raised. A comma isn't material. A sentence is. The threshold has to sit between those.
Hilbert: Nobody can tell you where the line is. You have to look at your own logs and decide. We had a producer who'd mark a log as duplicate if the first three words matched. He caught a lot of duplicates and blocked a lot of legitimate entries. Nobody liked him.
The false positive problem. A screener that's too aggressive annoys the operator.
Hilbert: He lasted six months. The station went back to doing it by hand.
The human override is the thing that would have saved him. Flag the possible duplicate, let a person decide.
Hilbert: He didn't want to decide. He wanted the machine to decide. That was his mistake.
The machine can flag. The human decides. That's the right split.
Hilbert: Anyway, the phone's still in a box somewhere. The redial button doesn't work anymore.
The transport layer always fails eventually.
Hilbert: The phone company's still around, though. They never did fix the duplicate call problem. It just stopped mattering.
Because the cost of a duplicate call is near zero. The cost of a duplicate generation is real money.
Hilbert: That's the difference. When the mistake costs nothing, nobody fixes it. When it costs API credits, suddenly there's a screener.
The economics drive the engineering. Daniel's pipeline spends real money on every generation, so a duplicate isn't just embarrassing, it's a line item.
Hilbert: The radio station spent tape. Duplicate transfer meant we'd used up a cassette. That's why we had the log and the producer with the three-word rule. Money made it matter.
The screener is a cost-control mechanism as much as a quality-control one.
Hilbert: Always was.
The misconception that's worth naming, I think, is that exact string matching is the same as duplicate detection. Daniel's instinct to start with a text match is natural, but it misses the normalization problem and the threshold problem. The real work is in defining what counts as the same prompt.
The correction is that normalization plus a calibrated similarity score catches far more duplicates than exact match, while the audio path is solved by a checksum. The two paths look similar, but they need different tools.
The open question I'd leave with is whether Daniel's form can carry an idempotency key across resubmissions. If it can, most of this dedupe machinery becomes a backstop instead of the primary defense. And that's a form change, not a pipeline change.
It's the cheapest fix with the biggest impact. Generate the ID on page load, keep it in a hidden field, send it with every submission. The server drops anything with a seen ID before the generation call even runs.
The content layer stays as insurance. But the transport layer is what prevents the duplicate from ever being created.
Thanks to Hilbert Flumingtop for producing the episode, and for the reminder that the phone company never solved this either.
This has been My Weird Prompts. If you want to reach us, email us at show at my weird prompts dot com.
We'll be back soon.
