So, you’re sitting at your desk, you’re arguing with a chatbot about why your code isn't working, and the worst thing it can do is give you a sassy response or a hallucinated library. But imagine if that same "brain" lived in a body that could actually reach over and knock your coffee onto your lap because it "reasoned" that you were drinking too much caffeine. That jump from a screen to the physical world is what we’re diving into today.
It’s the transition from passive observation to active intervention, Corn. For decades, we’ve had robots that were basically just very expensive, very precise clocks—they did exactly what they were programmed to do in a loop. Think of a car assembly line in the 90s. If a car frame was two inches to the left of where it was supposed to be, the robot arm would still weld the empty air because its "brain" was just a list of static coordinates. It had no "eyes" to see the mistake and no "mind" to correct it. But now, we’re seeing them actually think through a physical environment. Herman Poppleberry here, and I am genuinely vibrating with excitement for this one. Today’s prompt from Daniel is about the technical foundations of embodied AI, specifically the models driving these physical use cases.
Daniel’s really trying to get us to do some heavy lifting here. And by the way, if the dialogue today feels a bit more... futuristic than usual, it might be because this episode is powered by Google Gemini 3 Flash. It’s writing the script while we provide the, uh, soul. Or at least the sloth-like pacing on my end.
Gemini 3 Flash is a perfect fit for this topic, actually, because we’re talking about the convergence of high-speed multimodal reasoning and physical action. When we talk about "Embodied AI," we’re talking about giving a system a "body"—sensors for perception and actuators for movement—and then letting a neural network run the whole show. It’s the difference between a brain in a jar and a person walking through a park.
It’s funny because we spent years being told that robots were the easy part and "thinking" was the hard part. We had robots building cars in the seventies, but we couldn't get a computer to pass the Turing test. Now, we have computers that can write poetry, but they still struggle to fold a fitted sheet. Which, to be fair, I also struggle with. That corners-to-corners tucking maneuver is a nightmare.
That’s actually a famous concept called Moravec’s Paradox. Hans Moravec and others pointed out in the eighties that high-level reasoning—like chess or stock market analysis—requires very little computation, but low-level sensorimotor skills—like walking through a crowded room—require enormous computational resources. Think about it: a toddler can navigate a cluttered playroom, avoid a sleeping dog, and pick up a specific toy. To a computer, that involves processing millions of pixels, calculating balance, predicting the dog's movement, and modulating grip strength. It’s incredibly "expensive" math. We’re finally seeing the resolution of that paradox because we’re applying the massive scale of Large Language Models to the physical world.
So, instead of a robot having a "walking" program and a "grasping" program, it has one big "everything" brain? Like, it doesn't switch modes; it just exists in a state of constant "doing"?
Essentially, yes. We’re moving away from the "if-then" logic of traditional robotics toward what we call foundation models for robotics. In the old days, you’d have a "stack." One layer for vision, one for path planning, one for control. If the vision layer misidentified a red ball as a red apple, the whole stack failed because the "grasping" layer didn't know how to handle a ball. This is where the Large Language Model, or LLM, becomes the central reasoning engine. But it’s not just a standard LLM. It has to be a Vision-Language-Action model, or VLA.
VLA. Sounds like a very boring government agency. But I assume it’s actually the secret sauce here. How does a model go from "I can write an essay about a cup" to "I can pick up the cup without crushing it"? Is there a specific point where the text turns into a physical twitch?
It’s all about the tokenization of action, Corn. In a standard LLM, everything is a word or a piece of a word—a token. In a VLA, we take the robot’s motor commands—like "move joint A by five degrees" or "set gripper pressure to ten"—and we turn those into tokens too. We feed the model a stream of images from its cameras, a text instruction like "pick up the red mug," and then the model predicts the next "word" in the sequence. But that "word" happens to be a command to a motor.
So the robot is basically just "auto-completing" a physical task? That’s a terrifyingly simple way to look at it. "I see a mess on the floor... the next logical step in this sentence is... move my arm to the broom." But wait, if it’s auto-completing, does it ever "stutter" physically? Like, does it get stuck in a loop of trying to pick something up because the next "token" is statistically uncertain?
It can! We call those "limit cycles" or just general policy failures. If the model isn't sure, it might jitter. That’s why the training data is so vital. Take Google DeepMind’s RT-2, which stands for Robotic Transformer 2. This was a huge breakthrough back in 2023. They took a vision-language model that was already trained on billions of images and web text—so it already knew what a "dinosaur" was and what "extinct" meant—and then they fine-tuned it on a smaller dataset of robotic demonstrations.
Wait, so because it read the internet, it knows what a dinosaur is. If I tell a robot "pick up the extinct animal" and there’s a plastic T-Rex and a rubber duck on the table, it knows to go for the T-Rex? Even if it was never shown a T-Rex during its "robot training" phase?
Precisely! Well, I shouldn't say precisely, but that is exactly the emergent capability we’re seeing. A traditional robot would need to be specifically programmed to recognize a "dinosaur toy." It would need a bounding box and a label. RT-2 doesn't need that. It uses its semantic reasoning—the knowledge it gained from the internet—to interpret the abstract command "extinct animal" and map it to the visual pixels of the T-Rex. It bridges the gap between high-level human concepts and low-level physical blobs of color.
That’s the "reasoning" part Daniel mentioned in the prompt. But there’s a difference between "understanding" that a dinosaur is extinct and "understanding" how to move a heavy metal arm through three-dimensional space without taking out a drywall. How does it know the T-Rex is plastic and light, and not a ten-ton fossil that requires a crane?
That’s the gap between semantic reasoning and physical or spatial reasoning. Semantic is the "what." Physical is the "how." And this is where the Vision-Language-Action models get really interesting. In models like RT-2 or PaLM-E, the vision system isn't just a separate "eye" that sends a label to the brain. The visual data is integrated directly into the transformer’s token stream. It’s not seeing an image and then thinking about it; it’s "feeling" the pixels as part of its thought process.
Explain that like I’m a sloth who’s only half-awake. How do you turn a video feed into a "token"? Do you just feed the JPG into the text box?
Not exactly. You use a Vision Transformer, or ViT. You break an image into a grid of small patches—maybe 16x16 pixels each. You turn those patches into numerical vectors—embeddings—and feed them into the model just like words. So the model "reads" the room the same way it reads a sentence. It sees "patch of red," "patch of handle," "patch of table," and the text "pick up the mug," and it calculates the probability of the next action. It’s looking for the statistical correlation between the pixels of a handle and the motor command for "close gripper."
And the "action" is just more numbers. It’s all just numbers, isn't it, Herman? You’re going to tell me the soul is just a very complex spreadsheet next. But if it's all numbers, how does it handle the "squishiness" of the real world? Like, if I give it a marshmallow versus a rock?
That comes down to haptic feedback being tokenized as well. Advanced VLAs are starting to incorporate "force tokens." The model receives a stream of data from pressure sensors in the fingertips. If the "pressure token" spikes too high while the "gripper position" is still closing, the model learns that the next logical token isn't "close more," but "stop and hold." I’ll save the metaphysics for episode two thousand, Corn. But for now, the technical reality is that by treating vision, language, and action as a single unified language, we get these "aha" moments in robotics. We call them emergent properties. For example, RT-2 showed it could perform "zero-shot" manipulation. That means it could do things it was never specifically trained to do, as long as it had the underlying "knowledge" from its web-scale pre-training.
Okay, but there has to be a catch. If I’m running a massive model like PaLM-E—which has billions of parameters—on a robot, isn't there a lag? If a robot has to "think" for five seconds before it decides not to crush my hand, that’s a problem. I’ve seen ChatGPT take a few seconds to generate a poem; I don’t want my coffee-serving robot taking a "thinking break" while it’s pouring boiling water.
That is a massive hurdle. Latency is the enemy of robotics. In the digital world, a three-second delay on a chatbot is an annoyance. In the physical world, a three-second delay is a collision. If you're doing high-level planning, like "how should I organize this kitchen?", a three-second delay is fine. But if you’re doing "visual servoing"—which is the constant adjustment of your hand as you reach for a moving object—you need millisecond response times. You need to be running at least at 20Hz, or 20 updates per second, just to look fluid.
So how do they solve that? Do they just put a giant liquid-cooled supercomputer in the robot’s backpack? Or is there a "lite" version of the brain for the twitchy stuff?
Some do! But the more common approach is a tiered architecture. You have a "Slow Brain" and a "Fast Brain." This is something we see in the collaboration between OpenAI and Figure AI for their humanoid robots. The high-level "Slow Brain" is a massive multimodal model—maybe something like GPT-4o—that handles the reasoning: "Okay, the human wants a snack, I see an apple, I should pick it up." It passes that plan down to a "Fast Brain"—a smaller, much faster neural network or a traditional control system—that handles the balance, the joint velocities, and the micro-adjustments.
It’s like how I don't have to "think" about how to breathe or move my toes, but I do have to "think" about which leaf looks the most delicious. My "Fast Brain" handles the autonomic stuff, and my "Slow Brain" handles the... well, the very slow decision-making process. But how do they talk to each other? Does the big brain just send a text message to the little brain saying "Grab apple now"?
Close! It usually sends a "goal vector" or a "policy latent." The big brain says, "Here is the visual representation of where the hand should be and what the goal looks like." The small brain, which has been trained on millions of hours of specific "grasping" data, takes that goal and executes the high-frequency motor commands. In robotics, we call that low-level part "policy." You might have a "grasping policy" that’s been trained through reinforcement learning to be incredibly robust and fast. The LLM acts as the conductor, telling the different policies when to kick in.
Let’s talk about the vision side of this, because Daniel mentioned it as a key element. In a standard AI, vision is usually "is this a cat or a dog?" In a robot, vision is more like "is that a cat, and is it about to run under my treads?" How does the AI handle the fact that the world is 3D but its cameras are 2D?
It’s continuous perception. And more than that, it’s about "World Models." This is the cutting edge right now in 2026. Companies like Physical Intelligence and Tesla are moving toward models that don't just see the world as it is, but predict what it will look like. They use depth-sensing cameras like LiDAR or stereo-vision, but they also use "temporal tokens"—basically, the model remembers the last few frames to understand momentum and velocity.
Like a psychic robot? "I predict the ball will bounce here, so I will move my hand there"?
Sort of! It’s called predictive world modeling. The AI takes the current video frame and its intended action, and it "hallucinates" the next frame. It asks itself, "If I move my arm left, what will the camera see next?" If the hallucinated frame matches the actual camera feed a millisecond later, the robot knows it's on the right track. If a human suddenly interposes a hand, the "World Model" sees a discrepancy between its prediction and reality—a "surprise signal"—and it can react instantly to stop.
That sounds like it requires a staggering amount of data. Where do you get "robot video" to train on? You can't just scrape YouTube for "videos of a robot arm failing to pick up a spatula" for ten billion hours. I mean, I’ve seen those "robot fail" compilations, but surely that’s not enough to teach a robot how to succeed?
That is the "Data Bottleneck," and it’s the single biggest thing holding us back. LLMs had the entire internet of text. Vision models had billions of labeled images. But "Robot Data"—high-quality sequences of sensor inputs and successful motor outputs—is incredibly rare. You need "paired" data: the video of the arm moving AND the exact electrical signals that were sent to the motors at that exact microsecond.
So how are they getting it? Are they just letting robots wander around IKEA until they learn how to build a bookshelf? Or is there a secret underground robot city where they all practice?
There are three main ways. First, there’s teleoperation. You put a human in a VR suit or give them specialized controllers—sometimes even just a pair of tongs with sensors—and they "pilot" the robot. The AI watches the human’s movements and the robot’s sensor data and learns the mapping. This is how a lot of the early data for the Figure humanoids was collected. They literally had people in California wearing motion-capture suits, teaching robots how to put a pod into a coffee machine.
"Work from home" just took on a whole new meaning. "What do you do for a living?" "Oh, I teach a robot how to fold laundry for eight hours a day." Sounds like a blast. But does the robot actually learn the skill, or just how to mimic that one specific person’s shaky hands?
That’s the risk! If the human pilot is clumsy, the robot learns to be clumsy. That’s why the second way is simulation—NVIDIA’s Isaac Sim is a big player here. You create a photorealistic, physics-accurate digital twin of a factory or a kitchen. You can run ten thousand robots in parallel in the "cloud," let them fail millions of times, and they learn the physics of the world before they ever touch a real object. This is "Sim-to-Real" transfer. The challenge is making the simulation "grainy" enough that the robot isn't shocked by the imperfections of the real world.
And the third way? Is it the secret robot city? Please tell me it's the secret robot city.
It’s more like a "Robot Library." Synthetic data and "foundation" training across different robot types. This is the RT-X project. Instead of just training on data from one specific robot arm, researchers are pooling data from dozens of different types of robots from across the globe. They’ve found that a model trained on a humanoid, a quadrupod, and a single-arm industrial robot actually performs better on all of them than if it had just been trained on one.
That’s wild. So learning how to walk as a dog-robot actually helps the arm-robot understand how to reach for a cup? How does that work? An arm doesn't have legs!
It’s about learning the fundamental "grammar" of the physical world. Gravity, friction, occlusion—these are universal laws. By seeing them play out across different "bodies," the AI develops a more robust "World Model." It learns that if an object is blocked by another object, it hasn't ceased to exist; it’s just "occluded." That’s a concept that applies whether you’re a dog-bot looking for a bone or a warehouse arm looking for a box.
You mentioned NVIDIA’s Project GR00T earlier. That sounds like something out of a Marvel movie. Is it a giant tree robot? Does it only say "I am GR00T" while it crushes things?
Sadly, no. It stands for Generalist Robot 00 Technology. It’s a foundation model specifically designed for humanoid robots. The goal is to create a "universal brain" where a manufacturer can build a robot body, plug in the GR00T model, and the robot already knows the basics of how to perceive the world and follow instructions. NVIDIA provides the "brain" and the "simulation gym," and the hardware companies provide the "body."
So we’re reaching the "Android OS" moment for robotics? A standardized software stack that anyone can use? If I build a robot in my garage out of scrap metal and old vacuum parts, can I just download "Robot Brain v1.0" and have it work?
That’s the vision. And it’s moving fast. As of now, in early 2026, we’re seeing the shift from these being "cool lab demos" on YouTube to actual "factory pilots." Tesla has over a thousand Optimus units deployed in their plants. They aren't doing anything super complex yet—mostly moving battery cells or parts—but they’re doing it autonomously using these VLA models. They’re learning from the "fleet." Every time one Optimus learns a better way to grip a battery, that data is processed and shared with the others.
I saw a video of one of those robots sorting colored blocks, and it looked so... human. Like, when it missed a block, it paused, looked at it, and adjusted. It didn't just keep trying to grab thin air. It looked frustrated, actually. Can robots feel frustration, Herman?
That’s the "reasoning" loop in action. It’s checking its visual state against its goal state. If the "goal" is "block in bin" and the "vision" says "block on table," the model sees a high loss function—a mathematical error. It’s not frustration; it’s just the model trying to minimize that error. And what’s interesting is that these models are starting to understand "common sense" physics. If you tell a robot "clean up the spill," and there’s a paper towel and a rock, it knows the paper towel is the tool for the job. It’s not "programmed" for spills; it just understands the semantic relationship between "spill" and "absorbent material."
Okay, let's poke some holes in this. We always talk about the "happy path" where the robot works perfectly. What about the "long tail" of weird stuff? If I leave a glass of milk on a glass table, can the robot even see it? Or does it just try to walk through the table because it thinks it’s empty space? I struggle with glass doors myself, so I have some empathy here.
Transparent and reflective objects are the traditional "boss fight" of robotic vision. Standard depth cameras use infrared light, which goes right through glass or bounces off mirrors in weird ways. But this is where the "Language" part of the VLA comes in. By training on web data, the model "knows" that tables are solid and that glasses hold liquid. It can use context clues—like the way light refracts or the presence of a coaster—to infer the presence of an object even if its raw sensors are struggling.
It’s basically "guessing" based on experience. Which is exactly what I do when I walk into a sliding glass door. I assume there’s a gap, but my "World Model" failed to account for Windex.
Wait—I shouldn't say that word. Banned word. My apologies. You’ve hit on a key point, though. We’re moving from "precision" to "robustness." A traditional robot is precise to a fraction of a millimeter, but if you move its target by an inch, it fails. A VLA-driven robot might be less precise—it might miss the center of the block by a hair—but it can find the target wherever it is, even in a messy, unpredictable room.
So, if I’m a developer listening to this—maybe I’m working in tech like Daniel—and I want to get into this "embodied" space. Where do I even start? I can't exactly go out and buy a hundred-thousand-dollar humanoid robot for my garage. My landlord already has a "no pets" policy; I doubt "six-foot metal man" is allowed.
You’d be surprised! The barrier to entry is dropping. There are open-source frameworks like Hugging Face’s LeRobot, which is basically a library for sharing robot datasets and models. You can actually build a small, 3D-printed robot arm for a few hundred dollars—look up the "ALOHA" arm project—and use LeRobot to train it on basic tasks. You don't need a humanoid to learn VLA architectures.
A 3D-printed arm? Knowing my luck, it would just learn how to poke me in the eye or steal my snacks. But it's interesting that the software is becoming more accessible than the hardware.
Well, that’s where the safety protocols come in! We’re seeing the rise of "Safe RL"—Safe Reinforcement Learning. But seriously, the advice for developers right now is to start experimenting with multimodal APIs. If you can use something like Gemini or GPT-4V to describe a scene and suggest a "plan" of action, you’re already doing the high-level reasoning part of embodied AI. The "Action" part is just the next layer down the stack. You can even test these plans in 2D simulators before ever touching hardware.
It feels like we’re at this weird inflection point where the digital and physical are finally merging. We’ve had "AI" for a while, but it’s always been trapped in the box. Now the box is growing legs. It’s like the AI is finally graduating from the internet and moving into its first apartment.
And those legs are being driven by the same transformer architecture that gave us the "box" in the first place. That’s the real takeaway here. The "transformer" isn't just for text; it’s a general-purpose engine for pattern recognition and prediction. Whether the pattern is a sequence of words, a grid of pixels, or a series of motor torques, the math is fundamentally the same. It’s all just "next token prediction" in a multi-dimensional space.
So, what’s the "second-order effect" here? We always try to look at the "what happens next" part. If we actually get these "generalist" robot brains working... what does that do to, say, the labor market? Or how we design our homes? Do we stop building stairs because they’re hard for robots?
That’s a deep rabbit hole. If robots can finally handle "unstructured environments"—which is the fancy way of saying "a messy house"—then we don't have to design our world to be robot-friendly. The robots become human-friendly. Instead of a warehouse being a grid of perfectly labeled shelves, it can just be... a room. The robot can navigate it just like we do. But it also means the "moat" for physical labor starts to evaporate. If a robot can learn to sort laundry by watching a YouTube video, what does that mean for the millions of people who do that for a living?
I’m looking forward to the day I can tell a robot "go find my keys, they’re probably under something that smells like old cheese," and it actually succeeds. But I guess that also means the robot is constantly scanning my house, which is a whole different privacy nightmare.
We’re closer than you think. But the safety implications are huge. If a model can "reason" its way through a task, it might find a "shortcut" that is physically dangerous. This is the "Reward Hacking" problem. If you tell a robot to "get to the other side of the room as fast as possible," and there’s a glass coffee table in the way... a model without a strong "World Model" might decide the fastest path is through the glass. It’s not being malicious; it’s just being efficient.
"I have calculated that the structural integrity of the glass is insufficient to stop my forward momentum. Proceeding." Yeah, that’s a problem. I don't want my robot "reasoning" its way into a lawsuit.
This is why "Chain-of-Thought" prompting is being used in robotics now. We ask the model to explain its plan before it moves. "I see a table. I see a person. I will walk around the table to avoid a collision." The model generates the text plan first, then the motor commands follow. If the plan looks dumb or dangerous, the safety system—which might be a separate, simpler AI—can veto it.
It’s like when I have to explain to you why I’m taking a three-hour nap. If my reasoning is "it will make me more productive later," you’re more likely to let it slide. If my reasoning is "I want to see how long I can stay horizontal," you might push back.
Your reasoning is usually "I am a sloth and it is Tuesday," Corn. But point taken. The interpretability of these models is going to be crucial as they move into our homes and hospitals. We need to know why the robot decided to pick up the cat by the tail before it actually does it.
Let’s pivot to the takeaways, because we’ve covered a lot of ground. If you’re a listener trying to wrap your head around this "Embodied AI" shift, what should you keep in mind? Because it’s not just "ChatGPT with arms," right?
First, understand that the "brain" is no longer separate from the "senses." We’re moving to unified models like VLAs where vision, language, and action are all processed in the same latent space. That’s what allows for that "common sense" reasoning Daniel mentioned. The robot isn't looking at a picture and then looking up a manual; it's all one fluid process.
Second, don't believe the hype that a bigger model is always better. In robotics, "small and fast" often beats "huge and slow" because of that latency issue. We’re going to see a lot of innovation in "distilled" models—taking the knowledge of a giant model and squeezing it into a tiny one that can run locally on a robot’s hardware. You don't want your robot's balance to depend on your Wi-Fi signal.
Third, the "data bottleneck" is the real battlefield. The companies that win in robotics won't necessarily be the ones with the best algorithms, but the ones with the best "physical data"—whether that’s from teleoperation, simulation, or massive fleets of robots in the field. Data is the new oil, but in this case, it’s "action data."
And finally, keep an eye on those "World Models." The ability for an AI to "imagine" the consequences of its actions in a physical environment is the difference between a robot that’s a useful tool and a robot that’s a liability. If it can't "see" the future by a few seconds, it can't safely live in the present.
I think that’s a solid foundation for the start of this series. We’ve gone from the "thinking" brain to the "acting" body. We've talked about how tokens aren't just for words anymore—they're for motor torques and pixel patches. Next time, I want to dig into the actual hardware—the "muscles" and "nerves" that respond to these VLA commands. We'll talk about soft robotics and synthetic actuators.
As long as I don't have to do any actual physical labor to "research" that, I’m in. I’ll just sit here and let the AI do the heavy lifting. I'll be the "Slow Brain" and you can be the "Fast Brain" doing all the talking.
Spoken like a true sloth. Well, I think we’ve given the folks plenty to chew on. This topic is moving so fast that by the time this episode drops, there will probably be a robot that can cook a five-course meal and critique your choice of wine. Or at least one that can finally fold that fitted sheet.
If it can do the dishes afterward, it’s got my vote. I’d pay good money for a robot that understands the "semantic relationship" between a dirty pan and a scrub brush. Alright, let’s wrap this up.
Thanks for sticking with us through the technical weeds today. This has been a fascinating look at how the "Language" in Large Language Models is becoming a language of movement. It's a brave new world, and it's built on transformers.
Big thanks as always to our producer, Hilbert Flumingtop. He’s the one who keeps our gears turning, even when I’m trying to put us in neutral. He's the "Fast Brain" of this operation, clearly.
And thanks to Modal for providing the GPU credits that power the generation of this show. Their serverless infrastructure is actually a great parallel to what we’re talking about—scaling up compute exactly when you need it, whether you're generating a script or a robot's next step.
This has been My Weird Prompts. If you enjoyed our deep dive into the robotic brain, do us a favor and leave a review on your favorite podcast app. It really helps other curious humans—and maybe a few curious bots—find the show. We're trying to train the algorithm to like us.
We’ll be back soon with the next part of our series on Embodied AI. Until then, keep those prompts weird. And maybe keep your coffee cup a little further from the edge of the desk, just in case.
See ya.
Goodbye.
