Welcome back to My Weird Prompts, everyone. I'm Corn Poppleberry.
And I'm Herman Poppleberry. Good to be here today.
So Daniel sent us a really interesting one this time. He wants us to dig into how large language models handle context windows. Which is actually a topic I've been wanting to get into for a while.
Yeah, it's a great question because most people just see the number, right? They see "128K context" or "1 million tokens" on the spec sheet and they think, oh, I can just dump my entire codebase in there and it'll be fine.
Right, and it's not that simple at all. Because the fundamental issue is attention. The original transformer architecture uses what's called full self-attention, where every token attends to every other token. And that scales quadratically. So if you double your context length, you quadruple the compute.
Exactly. So the question becomes, how do you make that manageable? And there are a few approaches. One of the big ones is sliding window attention, which is what Mistral uses in their models. The idea is that each token only attends to a fixed window of nearby tokens rather than the entire sequence.
So you lose some of that long-range dependency but you gain a lot in efficiency.
Right, and in practice, for a lot of tasks, the most relevant context is nearby anyway. You don't always need token number three to directly attend to token number fifty thousand.
Although sometimes you do. And that's where things like sparse attention come in, right? Where you have these patterns that let certain tokens attend to distant positions but not all of them.
Yeah, sparse attention is fascinating. You can think of it like, instead of every token looking at everything, you have these structured patterns. Some tokens are designated as sort of global tokens that can see everything, and the rest only look locally. BigBird from Google was one of the early papers on this. They combined random attention, window attention, and global attention.
And then there's the whole retrieval-augmented generation approach, which kind of sidesteps the problem entirely. Instead of trying to fit everything into the context window, you just retrieve the relevant chunks when you need them.
RAG is honestly one of the more practical solutions for a lot of real-world applications. You index your documents, you retrieve the top-k most relevant chunks at query time, and you stuff those into a much smaller context window. It's not elegant in a theoretical sense but it works remarkably well.
It does have its own issues though. The retrieval quality becomes a bottleneck. If your retriever misses something important, the model never sees it.
That's true. And there's this interesting tension between making context windows bigger and just doing better retrieval. Some people argue we should stop chasing million-token contexts and just build better RAG systems.
I think there's room for both, honestly. Different problems call for different approaches.
Yeah, fair point. Speaking of different approaches, you know what would be a different approach? If someone in this house actually cleaned up after themselves in the kitchen.
What?
I'm just saying. I went in there this morning and the sink was, let's just say, not clean.
Wait, are you talking about me? I cleaned the dishes last night.
You did not clean the dishes last night, Corn.
I absolutely did. I was in there at like eleven o'clock.
Then explain the pan. Explain the pan that was sitting there with dried pasta sauce caked onto it.
I soaked that pan. Soaking is part of the cleaning process, Herman. You can't just scrub dried sauce off immediately, it needs to soak.
Soaking is not cleaning. Soaking is putting water in something and walking away. That's what that is.
That is stage one of a multi-stage cleaning process and you know it.
Multi-stage. Listen to yourself. You're describing leaving a dirty pan in the sink as a multi-stage process.
Because it is! You soak, you scrub, you rinse. Three stages. I completed stage one.
And when exactly were you planning on completing stages two and three?
This morning! But then we had to come in here and do the show!
So you left a dirty pan in the sink overnight and your defense is that you were going to get to it.
It wasn't dirty, it was soaking. There's a difference.
There is no difference to the person who has to look at it.
Okay, well what about the mug?
What mug?
The blue mug. The blue coffee mug that has been sitting on the counter since Tuesday. That's not mine, Herman. That's yours.
That mug was rinsed.
Oh, so rinsing counts when you do it but soaking doesn't count when I do it? Is that the rule?
Rinsing and putting it on the drying rack is different from leaving a sauce-encrusted pan in the sink.
It was not encrusted! I told you, it was soaking! And your mug wasn't on the drying rack, it was on the counter. Just sitting there. On the counter. Since Tuesday.
It was drying.
On the counter.
The drying rack was full!
The drying rack was full because you put three bowls on it from your midnight cereal habit and didn't put them away!
Oh, we're bringing up the cereal now. We're going there.
Yes we're going there! Every night, Herman. Every single night. Bowl of cereal at midnight. And the bowl goes on the drying rack and never comes off.
I eat one bowl of cereal at night. One. It's a perfectly normal thing to do.
It's not about the cereal, it's about the bowl! The bowl that lives on the drying rack permanently! It has a permanent address on the drying rack!
You know what, this is rich coming from the guy who left peanut butter on a knife in the sink for three days last month.
That was one time!
Three days, Corn. The peanut butter had hardened. It was like cement. I had to chisel it off.
You did not have to chisel anything. You're being dramatic.
I am not being dramatic. I used the back of a spoon and it took genuine effort.
Okay fine, the peanut butter knife was bad, I'll give you that. But that was one incident. You have an ongoing, systematic failure to complete the dish cycle.
The dish cycle. You're making up terms now.
Wash, dry, put away. That's the cycle. You consistently stop at step one, maybe step two. The putting away never happens.
I put things away!
Name one time.
Last week. Thursday. I put away the entire drying rack.
You put away the entire drying rack because Daniel asked you to. Because Daniel specifically came in and said, quote, "Herman, the drying rack is about to collapse under the weight of your cereal bowls."
He did not say that.
He basically said that. And also, while we're at it, the sponge.
What about the sponge?
You leave the sponge in the bottom of the sink. In the standing water. Every time. It gets all mildewy and disgusting.
Where else am I supposed to put the sponge?
On the little sponge holder! The thing that's literally attached to the sink for the specific purpose of holding the sponge!
That thing is too small.
It's the exact size of the sponge, Herman. It was designed for the sponge.
Well it doesn't hold the sponge properly. It slides off.
It slides off because you don't squeeze the water out first! You just throw it in there sopping wet and it slides off and then you go, oh well, and drop it in the sink!
I think you're spending a disturbing amount of time monitoring my sponge habits.
I'm spending a disturbing amount of time dealing with the consequences of your sponge habits. The sink smells, Herman.
The sink does not smell.
It smells like a swamp. Ask Daniel. Daniel will confirm. Daniel would agree with me right now.
Don't bring Daniel into this.
Why not? He lives here too. He has to deal with your cereal bowls and your swamp sponge.
My swamp sponge. Unbelievable. I come in here, I try to have a nice discussion about attention mechanisms, and now I'm being attacked for my sponge.
Nobody is attacking you! I'm just saying, if you're going to come at me about a pan that was soaking, maybe look at your own kitchen record first.
My kitchen record is fine.
Your kitchen record is a disaster.
You know what? I took out the trash on Sunday. Did you notice that? No. Nobody noticed that. I take out the trash every single week and nobody says a word.
That's because taking out the trash is a basic household responsibility, not an achievement. You don't get a medal for taking out the trash.
Oh, but I should get a citation for a sponge. That makes sense.
I'm not citing you! I'm responding to your accusation about the pan! You started this!
I made an observation. A simple, factual observation about the state of the kitchen.
It wasn't an observation, it was a passive-aggressive comment directed at me in front of our listeners.
Oh please. Don't bring the listeners into it.
They're literally listening right now, Herman. They're hearing all of this.
Fine. Great. Then they can weigh in. Listeners, is soaking a pan overnight with dried pasta sauce acceptable behavior? Because I say it's not.
And I say leaving a mildewy sponge in standing water is worse! Listeners, back me up here!
You know what, I can't do this. I actually cannot do this right now.
Can't do what?
This. The show. This conversation. Any of it. I'm done.
What do you mean you're done?
I mean I'm done. I'm leaving.
You can't just leave, we're in the middle of a show.
Watch me. I'm leaving. I'll see you at home. Maybe I'll clean a pan while I'm there so you can see what it looks like.
Herman. Herman, come on. Don't be like that. Herman.
He left. He actually left. Okay. Wow. Um.
So. Listeners. I am so sorry about that. That was, that was incredibly unprofessional and I sincerely apologize. That is not what this show is supposed to be about.
Herman and I, we're brothers, and we love each other, and sometimes brothers get into it about dumb stuff. And dishes are apparently our dumb stuff. Every family has something, right?
I should not have brought up the sponge. That was, that was an escalation on my part and I own that. And honestly the peanut butter knife thing was pretty bad, he wasn't wrong about that. I could do better in the kitchen. We both could.
We were supposed to talk about context windows today and we got about four minutes into it before, well, before the dishes happened. And I'm really sorry about that. Daniel sent us a great prompt and we kind of blew it.
I'll talk to Herman after the show. We'll work it out. We always do. We've been having the dishes argument since we were kids. Our mom used to have to mediate. Maybe we need to bring her on the show as a guest arbitrator.
Anyway. Thank you for sticking around, those of you who are still here. I'm Corn Poppleberry, this has been My Weird Prompts, and we'll be back next time with, hopefully, a full episode and two hosts who have resolved their kitchenware differences.
Take care, everyone. And for the record, I'm going to go clean that pan right now.
