Daniel sent us this one — he's got this habit, right? Every time he switches topics in ChatGPT, he manually starts a new thread. Got into it back when context windows were tiny, and now he can't shake it even though the windows are enormous. His question is: what if you could build an AI assistant that does this automatically? Either by detecting a topic shift and spawning a fresh thread, or — and he calls this the coolest implementation — routing your message to whichever past thread it actually belongs to. He wants to know if the first version is even feasible, what the engineering looks like, and whether you could just stitch it together with existing tools like LangChain.
The answer is yes, you absolutely can — and it's cheaper and faster than most people assume. But the reason Daniel's habit persists even with hundred-thousand-token windows is that large context windows don't solve the actual problem. They mask it.
The problem isn't "can the model technically hold more text." It's that every token of irrelevant history is noise the model has to attend to. There's a well-documented phenomenon where LLM response quality degrades as you stuff the context window with unrelated material — it's not a hard cutoff, it's a gradual dilution of focus. So Daniel's instinct to prune is correct even now. What's changed is that the pruning doesn't have to be manual.
We're not trying to fix a broken habit. We're automating a good one.
And the architecture for doing that is surprisingly straightforward. Let's walk through the lightweight version first — the pre-message classification hook.
Before you get into the plumbing, I want to sit with something Daniel said. He mentioned that people who don't understand how AIs work statelessly might not see the reason for starting new threads. That's a real gap in how most users think about these systems.
The model itself has no memory. Every API call is a blank slate. When you're in a ChatGPT thread, what's actually happening is the application is silently prepending your entire conversation history to each new message you send. That's it. Threads are a UI construct — OpenAI's own documentation is explicit about this. The model doesn't know threads exist. It just sees a big blob of text with your latest message at the bottom.
Which means "starting a new thread" isn't telling the AI to forget something. It's telling the application to stop sending the old stuff.
And when context windows were four thousand tokens — that was GPT-3 in twenty twenty — you had to be ruthless. A few long exchanges and you were out of room. Now we're at a hundred twenty-eight thousand tokens on GPT-4o and Claude three point five Sonnet, and Gemini one point five Pro supports up to two million tokens. You could paste the entire text of War and Peace into your context window and still have room for a conversation about your grocery list.
Which sounds like it should make thread management obsolete. But you said it doesn't.
It doesn't, for two reasons. One is the quality degradation I mentioned — the model's attention gets spread thin across irrelevant history. The second is cost. You pay for every token in that context window, every single message. If you've got a ten-thousand-token debugging session sitting in your history while you ask about dinner reservations, you're paying to send those ten thousand tokens to the API for absolutely no benefit. At scale, that's wasteful.
Daniel's dream of a single all-day thread is actually kind of a bad idea, even if it's technically possible now.
It's a bad idea if you implement it naively. But his proposed solution — auto-detecting topic shifts and spawning new threads — gives you the user experience of a single continuous conversation while keeping the context clean behind the scenes. The user never has to think about it. That's the elegant part.
Alright, let's open the hood. How does the pre-message classifier actually work?
Picture the pipeline. Every time the user hits send, before the message reaches the main LLM, it passes through a lightweight classifier. This classifier has exactly one job: look at the incoming message plus maybe the last few exchanges, and output a decision. "Same topic" or "new topic." Binary classification, maybe with a confidence score.
This has to happen fast enough that the user doesn't notice the delay.
That's the critical constraint. If your main LLM response takes two seconds and your classifier adds another half second, the user feels it. So you want this classification step to complete in under a hundred milliseconds, ideally under fifty. And you want it to cost basically nothing.
What are the options for actually building that classifier?
Two main paths. Path one is a tiny fine-tuned model — something like DistilBERT, which is a distilled version of BERT that's about forty percent smaller and sixty percent faster while retaining most of the accuracy. You'd fine-tune it on labeled examples of topic continuations versus topic shifts. Path two, and this is the one I'd reach for first, is using a small local LLM. Llama three point two one-billion-parameters can run on a laptop, no GPU required, and it can classify a message in under fifty milliseconds with zero API cost.
Zero API cost is a compelling number.
It's hard to beat free. But even if you went with a cloud option — say, GPT-4o-mini with a fifty-token classification prompt — you're looking at roughly zero point zero zero zero one dollars per call. At current pricing, that's fifteen cents per million input tokens. If you're a power user sending a hundred messages a day, your classification overhead is one cent per day.
Cost is basically a rounding error either way.
Cost is not the bottleneck. And this is where the local model approach shines — fifty milliseconds on-device versus two hundred milliseconds round-trip to an API. Over a ten-message conversation, that's half a second versus two seconds of cumulative classification overhead.
There's something I want to poke at, though. What happens when the classifier gets it wrong?
Two failure modes. False positive: the classifier thinks you've switched topics and spawns a new thread, but you were actually continuing. Now your follow-up lands in an empty context with no history, and the AI has no idea what you're talking about. False negative: the classifier thinks you're still on the same topic, so your message about dinner plans gets appended to a thread full of debugging logs, and now you're paying to send all that irrelevant history.
The false positive feels worse to me. Losing context mid-conversation is jarring.
It's worse for user experience, yeah. The false negative is worse for cost and quality, but the user might not notice immediately — they'll just get slightly worse responses over time. The way to handle this is threshold tuning. You don't just take the classifier's binary output. You look at the confidence score. If it's above, say, zero point nine for "new topic," you auto-switch. If it's between zero point five and zero point nine, maybe you surface a subtle confirmation — "Starting a new thread for this topic" — with an undo option. If it's below zero point five, you stay in the current thread.
You're essentially asking the user to be the fallback classifier.
Which is fine because the whole goal is to handle the obvious cases automatically. If you're debugging Python and suddenly ask "what's a good recipe for lasagna," that's an unambiguous shift. The classifier should nail that with near certainty. The ambiguous cases — like when you're discussing two related technical topics that could reasonably share a thread — those are the ones where you let the user decide.
Daniel specifically asked whether this could be implemented by joining up components in LangChain. What's the answer there?
LangChain gives you most of the scaffolding. Its ChatMessageHistory abstraction lets you programmatically manage conversation history — append messages, clear history, retrieve the full transcript. But here's the thing LangChain doesn't give you: a built-in topic shift detector. That's the piece you have to build yourself.
LangChain handles the boring plumbing, but the interesting part is custom.
You'd implement it as a custom callback or a RunnableLambda that sits between the user input and the chain execution. The flow looks like this: user message arrives, your classifier runs, if it says "new topic" you call clear on the ChatMessageHistory and start a fresh chain, if it says "same topic" you append to the existing history and continue. It's maybe fifty lines of Python.
Fifty lines is not a weekend project. That's a lunch break.
The classifier itself is the only non-trivial piece, and even that is mostly prompt engineering if you're using an LLM-based approach. Something like: "You are a topic continuity classifier. Given the last three messages in a conversation and the user's new message, determine whether the new message continues the same topic or starts a new one. Respond with only CONTINUATION or NEW_TOPIC and a confidence score." That's it.
I want to push on something. You said "last three messages." How did you pick three?
That's a tuning parameter. Too few and you miss context — a single message might not capture the thread well enough. Too many and you're adding latency and cost to the classifier. Three to five is the sweet spot I've seen in practice. You're giving the classifier enough signal without bloating the prompt.
What about edge cases? Someone says "also" or "by the way" — those are linguistic signals that might indicate a topic shift or might just be a conversational tic.
That's where the fine-tuning approach wins over pure prompt engineering. If you train a DistilBERT model on thousands of labeled examples, it'll learn to distinguish between "also, regarding that bug" — which is a continuation — and "also, completely unrelated but" — which is a shift. LLM-based classifiers can pick up on these cues too if your prompt is explicit about them, but the dedicated model will be more consistent.
Alright, so we've got the simple hook. But Daniel mentioned a cooler version — routing messages to the correct past thread rather than just starting a new one. That feels like a different beast entirely.
It is, and it's where this gets genuinely interesting. The simple hook is a binary decision: stay or split. The routing version is a retrieval problem. You're asking: "Which of my past conversations does this message belong to?
Which means you need to remember your past conversations in a searchable way.
You need a retrieval-augmented generation layer over your own chat history. Here's how you build it. Every time a thread is closed — either manually or by the classifier — you generate a summary of that thread. A few sentences capturing the topic, the key points, maybe any decisions or conclusions. You embed that summary using something like OpenAI's text-embedding-three-small, and you store the embedding in a vector database. Chroma is the easiest to get started with — it's open source, runs locally, and integrates cleanly with LangChain.
Now you've got a searchable index of every conversation you've ever had with the AI.
When a new message comes in, before you even run the topic classifier, you embed the message and run a similarity search against your vector DB. Retrieve the top three most similar past threads. Then your classifier has a richer decision to make: does this message belong to one of those three existing threads, or is it new?
If it matches an existing thread, you resurrect that thread's full history and append the new message.
The user experience is seamless. They say "what was that decorator syntax again?" and suddenly they're right back in the Python debugging thread from three hours ago, with full context. They never had to think about which thread that belonged to.
This is starting to sound like a personal knowledge management system disguised as a chat interface.
That's precisely what it is. And I think that's the direction personal AI assistants are heading. Not just answering questions in the moment, but maintaining a persistent, organized, retrievable model of everything you've discussed with them.
What's the latency hit on the retrieval step?
Embedding generation takes maybe a hundred milliseconds via API. The vector search is near-instant — Chroma can search thousands of vectors in single-digit milliseconds. So you're adding maybe a hundred fifty milliseconds total before the classifier even runs. If you're using a local embedding model, you can cut that further.
How quickly does this vector database grow?
A thread summary embedding is fifteen thirty-six dimensions — that's about six kilobytes per thread. If you have a thousand conversation threads, your entire vector index is six megabytes. You could store a lifetime of conversations on a floppy disk.
A floppy disk. There's a reference for the kids.
The point is storage is not the constraint. The engineering challenge is all in the retrieval quality. How do you make sure the right thread surfaces when the user's query is vague? "That thing we talked about earlier" — that's a hard retrieval problem.
Which is why the simple hook might be the smarter first step. Don't try to route to past threads. Just detect topic shifts and start fresh. The user can always search manually if they need to revisit something.
That's the pragmatic engineering take. Ship the hook first. It solves eighty percent of the problem with twenty percent of the complexity. The routing system is the aspirational version — build it when you've validated that the hook actually improves the user experience.
Let's talk about what happens to the user's mental model when this works. Daniel's whole habit is built around manual thread management. If you take that away, does the user lose something?
This is the knock-on effect I find fascinating. When you manually start threads, you maintain a mental map of what the AI knows at any given moment. You're aware of the context boundary because you created it. If the system handles this automatically, that awareness fades. The user might be surprised when the AI doesn't remember something — or worse, when it references something from a thread the user thought was closed.
You need some kind of transparency. A thread dashboard.
A simple UI that shows active threads, their topics, when they were last used, and how many messages they contain. The user should be able to see what the system is doing and override it — merge threads that were incorrectly split, split threads that should have been separated, manually route a message to a specific thread.
This is starting to sound like an email client. Folders, threading, routing rules.
That's actually a useful analogy. What we're building is essentially an intelligent email client for AI conversations. The classifier is your auto-sorting rule. The vector DB is your search index. The thread dashboard is your folder view.
Just like email, the automation is great until it misfiles something important.
Which is why the undo mechanism matters so much. Every automatic thread switch should be reversible with a single click. "This message was moved to a new thread — undo?" That one feature eliminates most of the risk.
I want to circle back to something Daniel mentioned about people who don't understand statelessness. There's a broader point here about how AI products hide their architecture from users, and sometimes that hiding creates confusion.
ChatGPT's thread model is a perfect example. The interface presents threads as if they're persistent conversation sessions with a single AI that remembers you. But under the hood, it's just prepending text. The illusion is useful — it makes the product feel more natural — but it also creates exactly the kind of misconception Daniel flagged. Users think the AI has memory when it doesn't.
If you build an auto-threading system on top of that, you're adding another layer of illusion. The AI now appears to remember across threads, to know which conversation a question belongs to. But it's still just clever text management.
Which is fine, as long as the system works. The problem is when the illusion breaks. When the classifier misfiles a message and suddenly the AI seems to have amnesia about something you just discussed. That moment of broken illusion is worse than never having the illusion at all.
The engineering challenge isn't just technical. It's about managing user expectations and designing graceful failure pattern.
The best AI products are the ones where the seams don't show. But when seams inevitably do show, they should be informative, not confusing. A good thread dashboard isn't just a power user feature — it's the safety net that makes the automation trustworthy.
Let's get concrete for anyone listening who wants to build this. What's the weekend prototype look like?
Step one: install Ollama and pull Llama three point two one-billion. Step two: write a fifty-line Python script that uses LangChain's ChatMessageHistory. Step three: write a classification prompt that takes the last three messages plus the new message and outputs CONTINUATION or NEW_TOPIC. Step four: wrap that in a RunnableLambda that clears history on NEW_TOPIC and appends on CONTINUATION. Step five: build a minimal Streamlit UI that shows the current thread and lets you chat. That's it. You can build this in an afternoon.
The thread-routing version?
That's a weekend project, not an afternoon. Add Chroma for vector storage, generate summaries of closed threads using a cheap LLM call, embed them with text-embedding-three-small, and add a retrieval step before classification. The retrieval step searches for similar past threads, and the classifier decides whether to route to an existing thread or start new. More moving parts, more failure pattern, but magical when it works.
What about existing products? Is anyone doing this already?
ChatGPT has a search conversations feature, but it's manual — you have to know you're looking for something. Gemini has context caching, but that's more about reusing computed state than routing messages. The Assistants API from OpenAI has thread management built in, but it doesn't auto-detect topic shifts — you'd need to add that as a custom function call. What Daniel's describing is more proactive than anything I've seen shipped.
Which means there's an opportunity here. This isn't a solved problem.
It's not. And it's the kind of problem that's perfect for a solo developer or a small team. The core logic is simple. The value is in the integration and the user experience. You don't need a research lab to build this.
One thing we haven't touched on: privacy. If the system is archiving threads and storing embeddings, where does all that live?
If you're building a personal assistant, it should all be local. The vector DB runs on your machine. The embeddings can be generated locally too if you use a model like all-MiniLM. The only thing that leaves your device is the prompt to the main LLM — and even that could be routed through a local model if you're privacy-sensitive. There's no technical reason this needs to phone home.
If you're building this as a service for others, the privacy question gets thornier.
Now you're storing users' conversation histories, their thread summaries, their embeddings. You need encryption at rest, access controls, data retention policies. The engineering gets more complex not because the ML is harder, but because the operational responsibilities multiply.
For a personal project, keep it local. For a product, hire a security engineer.
That's the short version, yes.
I want to zoom out for a second. We've been talking about thread management as an engineering problem. But there's a philosophical question underneath it: how much should the AI manage versus how much should the user manage?
This is the tension in all of personal AI. The dream is an assistant that anticipates your needs, organizes your information, reduces your cognitive load. But every time the assistant makes a decision on your behalf, you lose a little bit of awareness and control. The art is finding the right boundary.
Daniel's prompt is really about that boundary. He wants to stop thinking about threads entirely. But he also wants the system to be correct enough that he doesn't have to think about it.
I think the threshold for "correct enough" is actually reachable with current technology. Topic shift detection is not a hard AI problem. It's a classification task with clear signals. The edge cases are manageable. The failure pattern are recoverable. This isn't like trying to build a general reasoning system — it's a focused, well-scoped problem.
What's the one thing a listener should take away from this?
If you're a power user of ChatGPT or Claude, pay attention to when the AI starts losing focus or hallucinating in long threads. That's your signal that you should have started a new thread — and it's also the signal a classifier would use. Notice those moments. Then, if you're a builder, prototype the classifier hook. It's cheap, it's fast, and it solves a real problem that even the big AI companies haven't fully addressed yet.
If you're not a builder, just knowing that threads are a UI trick, not real memory, will make you a more effective user. You'll understand why the AI sometimes seems to forget things, and you'll know when to give it a clean slate.
The best AI users are the ones who understand what's actually happening under the hood. Daniel clearly does. The rest of the world is catching up.
Alright, one open question before we wrap. How do you handle conversations that naturally span multiple topics? Like planning a trip — you're talking about flights, then hotels, then restaurants. Should the system split those into sub-threads or keep them together?
That's the hardest classification problem in this whole space. Those topics are related — they're all part of "trip planning" — but they're also distinct enough that the context from flights might not help with restaurant recommendations. I think the right answer is to keep them together under a parent thread but let the system recognize subtopics. That's a level of sophistication beyond what we've discussed today, but it's the natural next step.
As context windows keep growing — we're heading toward ten million tokens, not just two million — does thread management eventually become obsolete?
The technical need might diminish, but the cognitive benefit won't. Humans think in organized chunks. We don't want a single infinite conversation any more than we want a single infinite document. Structure helps us think. The best AI assistant won't just remember everything — it'll help you organize your thinking. And that's really what Daniel's asking for.
Good place to land. And now: Hilbert's daily fun fact.
Hilbert: In the late Victorian period, linguists believed that ergativity — the grammatical pattern where the subject of an intransitive verb is treated like the object of a transitive one — had gone completely extinct in Indo-European languages, until Basque was rediscovered as a living ergative language, having survived in isolation in the Pyrenees while every related structure vanished around it.
Basque grammar was hiding in a mountain range while the rest of Europe moved on.
This has been My Weird Prompts. If you enjoyed this episode, tell a friend who's still manually managing their ChatGPT threads — they need to hear it. You can find us at my weird prompts dot com. We're back next week.
