I have to be honest, my first reaction to this one was a kind of slow, dawning recognition. Like watching someone point at a load-bearing wall and ask why nobody talks about it.
And my first reaction was excitement, because this is a underexplored question. Daniel's asking why the people who study language for a living aren't in the room where language technology gets built.
Right. So Daniel's whole thing here, let me put it in his words as much as I can. He starts from the observation that large language models are what the world has decided to call AI, even though they're just a subset of transformer-based predictive models. He mentions we've talked before about how within the purest text-in, text-out task, you've got models that are far from legacy, like Google's BERT series. What strikes him is how much of that early work grew out of natural language processing and was spearheaded or collaborated on by linguists and linguistics bodies.
Which makes complete sense, he says.
Which makes complete sense, in his words. If large language models are fundamentally a technology about generating units of language, then linguistics is the natural home for the expertise that could push the frontier. But today, in mainstream discourse, we don't get the feeling linguists are at the forefront or closely involved in developing major new models. And what he wants to look at is the historical association between AI development and scholars and institutions whose primary remit was studying human language, and how that research got directly transferred into these models.
So the question has two halves. What was the relationship, and what happened to it.
And there's a third thing hiding in there, which is the implication that something was lost. So let's start with what these models actually are, and then trace where the linguistic DNA is hiding in plain sight.
The definition matters here because Daniel's right that the terminology has gotten sloppy. Large language models are a subset of transformer-based predictive models. The transformer architecture itself is from Vaswani and colleagues in 2017, "Attention Is All You Need." That's the substrate. BERT, GPT, RoBERTa, XLNet, all of those are variations built on top of it.
And BERT specifically is the canonical text-in, text-out model. Google, 2018. Bidirectional, masked-language-model pretraining. It's the one Daniel calls far from legacy, and he's right, it's still doing real work in classification, retrieval, a lot of the unglamorous stuff that actually ships.
The survey framing is useful here. Pretrained foundation models are the foundation for downstream tasks across modalities. BERT learns bidirectional encoder representations. GPT uses the transformer as a feature extractor and trains autoregressively. Same architecture family, different training objectives, different jobs.
So here's the tension we're going to sit in for the next twenty minutes. If these are language technologies, and they are, then why did the people who study language become peripheral to their development?
That's the episode. The hidden linguistic DNA in the architecture, why linguistics expertise is naturally central, what changed, and where re-engagement is happening now.
The first place to look is the architecture itself, because the transformer didn't come out of nowhere.
This is the part that surprised me when I first came across it. The transformer's core design idea came directly from linguistic syntax trees. There's a Google researcher, Jakob Uszkoreit, who initiated the transformer project. His goal was a more efficient, more scalable sequence model to replace recurrent networks and actually exploit GPU parallelism.
And his insight was about the structure of language itself.
His insight was that although language appears sequential, it's actually part-parallel, part-hierarchical. And that's exactly what linguists' sentence parse trees represent. You've got branches at each level that are independent of each other, and they combine at the next level up. That structure is what gave rise to the idea of a stack of parallel processing layers. That's what transformer layers are.
So the architecture that now dominates AI is, at its origin, a bet that linguists had already drawn the right picture of how sentences work.
And it's his own account, well-corroborated in interviews. But here's the detail that makes it almost too neat. Jakob Uszkoreit is the son of Hans Uszkoreit, who is a prominent computational linguist.
The family lineage is right there. The man who initiated the transformer grew up in a household where computational linguistics was the dinner conversation.
Attention itself built on Bahdanau and colleagues' work in recurrent sequence-to-sequence machine translation. So the whole thing emerged from the machine translation and NLP lineage. Not from some general abstract AI lineage. From people trying to translate sentences.
Which reframes the whole history. This wasn't AI people happening to work on language. This was language people building the thing.
And that goes back further than the transformer. NLP grew out of computational linguistics, which was itself a collaboration between computer scientists and linguists. Early NLP, from the fifties through the nineties, was heavily rule-based and grammar-driven. Chomskyan generative grammar, parse trees, formal syntax. Linguists weren't adjacent to that work. They were central to it.
So when did the center of gravity move?
Two revolutions. The statistical revolution starting in the nineties, and then the neural revolution in the twenty-tens. The transformer in 2017 and BERT in 2018 marked the point where large-scale statistical learning decisively displaced hand-coded linguistic rules.
And Daniel's intuition, that linguistics is the natural home for this expertise, turns out to be validated by the research. This isn't just a nice idea about who should be in the room.
It's validated in a few specific ways. Linguistic minimal pairs are now used to probe LLM internal representations. There's a study that ran over a hundred LLMs across a hundred and fifty thousand minimal pairs and found that LLM linguistic similarity aligns strongly with fine-grained theoretical linguistic categories, but weakly with broader ones.
Meaning the fine-grained distinctions linguists spent decades drawing are the ones that actually show up in the model's behavior.
The fine-grained ones are doing real analytical work. The coarse ones, not so much. That's a interesting result because it suggests the model has internalized distinctions that only a specialist would have thought to name.
Sure. Think about something like the difference between a raising verb and a control verb. "John seems to be happy" versus "John wants to be happy." On the surface they look nearly identical. Subject, verb, infinitive. But in raising constructions, the subject is semantically tied to the embedded clause, and in control constructions it's tied to the matrix verb. Linguists drew that distinction because of subtle asymmetries in how the sentences behave under passivization, under expletive insertion, under idiom chunks. "The cat seems to be out of the bag" works. "The cat wants to be out of the bag" doesn't, unless you mean something very different. Those are the kinds of distinctions the minimal-pair studies are finding the models have internalized.
So the model has, in some functional sense, learned the raising-control distinction without anyone teaching it the rule.
Without anyone teaching it the rule. And that's exactly why the linguistic framework matters. If you don't have the vocabulary to describe that distinction, you can't probe for it, and you can't tell whether the model has it or is just pattern-matching on surface statistics.
What about the interpretability side?
Hewitt and Manning in 2019 showed that syntactic distance and depth are embedded in BERT and ELMo word representations. That's a direct transfer of syntactic theory into interpretability work. You couldn't do that probe if you didn't already have a theory of what syntactic structure is.
And there's the brain alignment work.
AlKhamissi and colleagues found that LLM alignment with human language-network activity tracks formal linguistic competence, knowledge of linguistic rules, more closely than functional competence. Again, the linguistic-theoretic framing is doing the explanatory work.
So the DNA is in the architecture, it's in the representations, it's in the evaluation methods. The linguistics never actually left. It just stopped being credited.
That's a sharp way to put it. Though I'd add one nuance, which is that some of this is contested. The Uszkoreit account is his own, and it's well-corroborated, but it's a story about intent, not a formal proof that the parse-tree insight was necessary. You could imagine someone arriving at stacked parallel layers from a purely engineering direction.
Fair. But the family detail is hard to wave away.
The family detail is hard to wave away.
So the linguistics is in the DNA. Then why does it feel like the people who study language got shown the door?
This is where the literature gets interesting, because there are three distinct positions, not two. There's a paper from June of this year that frames it as insulationism, eliminativism, and conciliationism.
Walk me through them.
Insulationism says LLMs are irrelevant to linguistics. That's Chomsky and colleagues, Kodner and colleagues, Fox and Katzir. Their argument is that LLMs need trillions of tokens where children need under a hundred million words, they learn impossible languages as easily as natural ones, and they predict without explaining. The analogy they reach for is Ptolemaic epicycles. You can fit the data, but you haven't found the mechanism.
And Chomsky's New York Times piece from 2023 is the flagship statement.
"The False Promise of ChatGPT." The claim is that LLMs are not about natural language. That's the insulationist line in its purest form.
Then there's the opposite pole.
Eliminativism. Piantadosi in 2024, "Modern language models refute Chomsky's approach to language." And Ambridge and Blything, "Large language models are better than theoretical linguists at theoretical linguistics." The argument is that if LLMs acquire syntax without explicit rules or Universal Grammar, then the poverty-of-the-stimulus argument weakens. Maybe you don't need the innate machinery.
That's a strong claim. "Better than theoretical linguists at theoretical linguistics" is not a modest title.
It's not. And I'd flag that as contested. But the underlying observation, that models acquire syntactic competence from data alone, is hard to dismiss.
And the third position.
Conciliationism. LLMs are useful tools for testing and refining linguistic theory. Millière, and Futrell and Mahowald with the wonderful title "How linguistics learned to stop worrying and love the language models." The idea is that you use the model as an instrument, not as a replacement for theory and not as something to ignore.
So the "linguists are absent" story is really a story about which of these three positions won the institutional argument. And the answer is none of them, because the institutions stopped asking the question.
That's the shift. The dominant industrial labs, OpenAI, Google DeepMind, Anthropic, Meta, are engineering and scaling driven. The frontier moved from theory-driven NLP to compute and data-driven pretraining. Linguists' contributions became infrastructural and downstream rather than central.
Downstream how?
Data annotation and RLHF labeling is the clearest case. There are job postings that explicitly recruit software engineers, linguists, and writers to help shape the future of AI, labeling datasets and fine-tuning large language models, at around fifty dollars an hour. Linguists are in the loop. But as labor, not as architects.
Fifty dollars an hour to be the person who knows the difference between a dialect and an error, and you don't get to decide what happens with that knowledge.
You get to click acceptable or unacceptable. Then the engineers aggregate your clicks into a reward signal and ship.
There's something grim in that.
There is. And it's not just annotation. Evaluation and interpretability, the linguistic minimal pairs, BLiMP, the benchmarks. Low-resource languages, where you've got datasets like a Maithili sentiment corpus curated and validated by linguistic experts, or a Uyghur dependency treebank. All of that is linguistics doing necessary work at the edges of the pipeline.
Necessary but not load-bearing. Not in the room where the architecture gets decided.
And here's where the knock-on effect get interesting. If the people who study language are downstream of the technology that generates language, what does the technology miss?
The multilingual gap is one answer.
Performance drops up to twenty-four point three percent in low-resource languages on MMLU-ProX. That's not a rounding error. That's the technology working less well for most of the world's speakers.
And the sociolinguists are saying something similar about flattening.
There's a 2026 dialogue with five sociolinguists on GenAI's effect on linguistic diversity. Their warning is that it marginalizes minoritized varieties and flattens nuance. They're calling for inclusive co-design in GenAI development.
Which is a polite way of saying the models sound like one particular kind of English speaker and treat everything else as noise.
And there's evidence for the surface-level pattern matching too. LLM surprisal fails to predict garden-path and syntactic-ambiguity processing difficulty. Which means the model's internal statistics don't match how humans actually get confused by ambiguous sentences. It's doing something, but it's not doing what we do.
The models are good at the easy stuff and bad at the hard stuff, and the people who could tell you why are being paid by the item.
There's a counterweight though. And I want to be fair to the models here. GPT-4.5 predicted collective social-norm judgments better than one hundred percent of individual human participants. Gemini 2.5 Pro beat ninety-eight point seven percent, GPT-5 ninety-seven point eight, Claude Sonnet 4 ninety-six percent.
Better than every individual human.
But all of them showed systematic, correlated errors. So they're not just averaging human judgment. They're making the same mistakes in the same directions, which is exactly what you'd expect from a system that learned from a particular slice of text.
Same mistakes, same directions. That's the flattening the sociolinguists are warning about, showing up as a measurable error correlation.
That's the connection, yes.
Is there a re-engagement story, or is this just a eulogy?
There's a real re-engagement story, and it's more interesting than I expected. Cultural Interpretability is one. Jones, Satran and Satyanarayan propose a conjoint field of linguistic anthropology and machine learning, arguing that LLM interpretability and linguistic anthropology have convergent interests. Three axes, relativity, variation, indexicality.
Indexicality being the thing where a word points at social context, not just referent.
The goal is to improve value alignment with stylistically diverse speakers and culturally diverse speech communities. That's linguistics being pulled back into the alignment conversation, not as annotation labor but as theory.
What else?
The BabyLM Challenge. Warstadt and colleagues building data-efficient language models on child-scale input, under a hundred million words, to test cognitive and developmental plausibility. That's a direct bridge between developmental linguistics and model design. You're asking whether a model can learn language the way a child does, which is a linguistic question, not an engineering one.
The modal modeling framework.
Suzuki and Sugawara argue that LLMs have genuine epistemic value as minimal models, providing how-possibly explanations of language, even if they don't yet give how-actually explanations. They propose a continuum view rather than a binary.
The pendulum might be swinging back.
It might. But I'd want to be careful about the word "might." The re-engagement is happening in papers and in interpretability research. Whether it changes who builds the next frontier model is a different question, and I don't know the answer.
That's the honest place to leave it. The linguistics never left the architecture. It left the org chart.
The org chart is where the decisions get made.
Hilbert, you've been quiet back there, but I can see you've got something.
Hilbert: The annotation desk is where it happened first.
Say more.
Hilbert: Before any of this, before the big models, there was a company building what they called a grammar-aware writing assistant. I did a stint as a contract annotator. The job title was linguistic data specialist. That was a fancy way of saying I spent eight hours a day tagging whether sentences sounded right.
And the engineers?
Hilbert: The engineers kept calling the annotation guidelines "the grammar." I kept telling them there is no single grammar. There are registers. A sentence that's wrong in a legal brief is fine in a text message. They kept shipping anyway.
The expertise was there. It just wasn't allowed to change the spec.
Hilbert: The moment the work got broken into microtasks and paid by the item, the expertise stopped mattering. You can know everything about register and still be reduced to clicking acceptable or unacceptable on a single sentence with no context.
That's the structural point, isn't it. The pipeline design determines whether the expert's judgment can propagate.
Hilbert: The reason the models sound so confidently generic is that the people who could have told them "that's not how anyone talks" were the ones being paid by the hour to click. You don't get to say "that's not how anyone talks" when the task is one sentence at a time.
It's not that linguists got pushed out at the frontier. They got pushed out at the desk.
Hilbert: The frontier just inherited the result.
Did you keep any of it?
Hilbert: I kept the style guide. Spiral-bound. About forty pages of rules and one page that said when in doubt, ask a linguist.
One page.
Hilbert: One page. And nobody ever did.
Forty pages of rules and a single page of humility.
Hilbert: I also got fired for writing "this is not a grammar error, it is a dialect" in the comments field one too many times.
How many times is one too many?
Hilbert: Apparently four.
That's a good place to land, because it raises the question of what else got left at the annotation desk. If the transformer was born from a parse-tree insight, what other linguistic insights are sitting unused because the people who hold them aren't in the room?
The multilingual gap, the twenty-four percent drop in low-resource languages, the sociolinguists' warning about flattening nuance. The cost of excluding linguistic expertise isn't academic. It's baked into who the technology works for.
Which means the question isn't whether linguists should be involved. It's whether the field can afford to keep treating language as a dataset rather than a subject.
That's the open question. And I don't think we've answered it.
We haven't. Thanks to Hilbert Flumingtop for producing. This has been My Weird Prompts.
If you're enjoying the show, leave us a review wherever you listen. It helps.
We'll be back soon.
