Okay. I have read this prompt three times and I still don't know whether to laugh or check if my own calendar was ever managed by a person in a call center.
That's the correct response, honestly.
Daniel wants the most ridiculous, outrageous, well-documented cases of products sold as artificial intelligence where it turned out humans were quietly doing the intelligent part. He names two starting points specifically. X dot A I's Amy, the scheduling assistant. And Remotasks, the data labeling platform. He says we touched on this before and didn't do either of them properly, which is fair.
It is fair.
And then the two questions underneath it. How far has the illusion actually gone? And what is the single most absurd case of somebody believing they were talking to sophisticated AI and finding a human behind the curtain?
I have an answer to that second one and it is not the one most people would guess.
Then let's pull the curtain back. Start with the one that fooled Silicon Valley.
Amy.
Amy.
So the framing first, because it matters. There's a real distinction between human in the loop and human behind the curtain. Human in the loop is transparent. You know a person reviews the output. Radiology, for instance. The model flags, the radiologist signs off, everybody knows that's the workflow.
Right.
Human behind the curtain is different. The product is sold as autonomous. The human is not disclosed. And the person on the other end has been told, implicitly or explicitly, that they're talking to a machine.
So the deception isn't that humans are involved. It's that the involvement is hidden.
There's a name for it now. Fauxtomation. Automation that isn't automatic.
And this isn't three startups in a garage. It runs across scheduling, drive-thrus, retail checkout, data labeling. The common thread is that venture capital pays a premium for the word AI and pays nothing for the words we hired people.
Which is the whole incentive structure in one sentence.
So. Amy.
X dot A I launched Amy around twenty sixteen. The pitch was beautiful. An AI assistant that reads your email, negotiates times with other people's calendars, and books the meeting without you touching it. You cc Amy, Amy handles the back and forth.
And the back and forth is the hard part, right? That's not a lookup. That's negotiation.
That's negotiation with a stranger who has their own constraints and their own tone. And the reporting at the time, from Bloomberg among others, was that X dot A I brought in human contractors to handle the harder threads. People who would read the email chain and write the reply.
So Amy was sometimes a person named Amy.
Sometimes a person who was not named Amy at all.
And the persona did a lot of work there. The name, the little signature, the slightly formal phrasing. You're not suspicious because Amy sounds like an assistant, not like a system.
The delays helped too. If a reply took twenty minutes, that read as thinking. If it had come back in four seconds, you'd assume a script.
So the latency was the costume.
The latency was the costume. And that's the part I find clever, in the worst way. They didn't have to fake intelligence. They just had to fake the pace of it.
Did it hold?
It held long enough to raise a lot of money and long enough to become the example everybody cites. X dot A I pivoted away from the consumer assistant eventually. But Amy is the template. Every case we're about to talk about is Amy with a different costume on.
Okay, but hold on. I want to push on this a little, because I can already hear the objection. If a human is reviewing the output and fixing the mistakes, isn't that just a normal product development cycle? Every company does that. You ship a beta, you watch where it breaks, you patch it. What makes Amy different from any early-stage product with a support team?
That's a fair push and the distinction is disclosure. A beta with a support team is disclosed as a beta with a support team. You know there are humans. With Amy, the human was presented as the machine. The customer wasn't told they were interacting with a person at any point. That's the line. Not that humans were involved, but that their involvement was the product's central secret.
So the difference is consent.
The difference is consent. If you know a human might read your email, you write differently. If you think a bot is reading it, you write differently too. The company was harvesting the second behavior while delivering the first.
That's a good distinction. Okay, Remotasks. This one is different, because it's not a fake product. It's the supply chain underneath everybody else's real product.
That's the right way to put it. Remotasks is a data labeling platform. Workers annotate images, transcribe audio, moderate content, tag text. And the reason it matters here is that a model doesn't learn to see without somebody telling it what's in the picture.
Thousands and thousands of times.
Millions of times. And the labeling is done by people, largely in Kenya, India, the Philippines, Venezuela. The pay is per task and the per task rate is brutal. We're talking fractions of a cent for some task types.
A penny a task.
A penny a task is not unusual. And the worker has no benefits, no job security, no credit. Their name appears nowhere. When the model gets praised for understanding something, the understanding was taught by somebody making a penny at a time.
So the connection to fake AI is this. When a company says their product learns or understands, a huge amount of what looks like machine intelligence is actually human judgment that got baked in earlier and then frozen.
Compressed, yes. The labor is real. It's just upstream and invisible.
And there's a second layer, which is that some of these platforms have their own quality control done by humans pretending to be automated checks.
Right, and that's where it gets almost funny. You have humans reviewing the work of other humans so that a dashboard can report a percentage that suggests a machine did the reviewing.
A human pretending to be a machine checking the work of a human pretending to be a machine.
That's the whole industry in one sentence.
So why do companies do this? Because it can't only be cynicism.
It's mostly cost and reliability. A human handles the weird case correctly on the first try. A model in twenty twenty, twenty twenty-one handles the weird case by confidently inventing something. If your product is a scheduling assistant and it books a meeting for a date that doesn't exist, you have a customer service problem.
So the human is the error handling.
The human is the error handling. And the marketing decision is separate from the engineering decision. The engineers build the hybrid. The marketing department calls the hybrid AI.
And nobody in the room is technically lying, which is the part that should worry people.
Because human assisted AI is a true description. It's just that the assistance is doing most of the work and the AI is doing the greeting.
Give me a concrete version of that. What does the weird case actually look like for a data labeler?
Sure. Imagine you're labeling images of street scenes for an autonomous vehicle dataset. Most frames are easy. Car, pedestrian, traffic light. The model pre-labels those and you confirm. Then you get a frame where a person is carrying a large mirror down the sidewalk. Is that a pedestrian? Is that a reflective surface? Is it two pedestrians because of the reflection? The model has no idea. It labels it pedestrian with high confidence and it's wrong. You're the one who has to catch it, reclassify it, and write a note explaining why. That note is what teaches the next version of the model. You just spent six minutes on one frame that pays you four cents.
And that six-minute frame is the one that matters.
That six-minute frame is the entire reason the model gets better. The easy frames were never the problem.
Now. Presto.
Presto Automation. This is the one where the mask slips in the most entertaining way.
Set it up.
Presto sold a voice agent for fast food drive-thrus. You pull up, you order, the voice on the other end takes your order. And the pitch to investors was AI. Autonomous voice AI handling the order.
And the reality?
The Verge reported in December twenty twenty-three that a substantial share of those conversations were being handled by workers in the Philippines. People sitting at terminals, listening to the drive-thru audio, typing the order.
So the customer is talking to a person who is pretending to be a machine that is pretending to be a person.
And here's the detail that I think is the best single fact in this entire episode. The workers were reportedly instructed to avoid saying um or uh.
To sound more like AI.
To sound more like AI. Which means the training was, in effect, don't sound human. Strip the natural disfluencies out of your speech so the customer believes the smoothness is a machine.
And it worked. Customers didn't complain that the voice sounded like a person. They complained when it sounded like a bad person.
How many stores?
Presto had deals with a number of chains. The scale was real. And the disclosure was not. The word AI was doing the work in the investor deck and the human was doing the work in the drive-thru.
And then the legal trouble.
Presto ran into securities issues. There were questions about what had been represented to investors about the technology. The company's fortunes turned hard.
Which is the pattern. The technology is fine. The representation is the liability.
That's the sentence I'd put on the wall. The technology is fine. The representation is the liability.
Alright. So that's the pattern. Now the most absurd case of all.
Amazon Just Walk Out.
Just Walk Out.
The pitch was the cleanest thing Amazon has ever sold. You walk into a store, you take what you want, you walk out. No checkout, no scanning, no line. Cameras and sensors track what you picked up and charge your account.
Computer vision at scale.
Computer vision at scale. And Ars Technica reported in April twenty twenty-four that the system required roughly one thousand workers in India to manually review transactions.
A thousand.
On the order of a thousand. Watching video, checking what was taken, confirming the charge.
So the store with no checkout had a checkout. It was just in another country.
It was in another country and it was a person.
And the number is what gets me. One thousand people to eliminate the cashier.
The cashier was not eliminated. The cashier was relocated and the customer was told the cashier was a camera.
Amazon ended the system in April twenty twenty-four. Did they end it because of the labor, or because of the economics?
Both, and I'd be careful there. The public reporting frames it as the human review requirement becoming a problem. But Amazon has said the technology works and they're deploying it in other formats. There's a real argument that the issue was the cost per store at small scale rather than the concept.
So the honest version is the labor requirement was a big part of it and not necessarily the only part.
That's the honest version.
But the headline fact stands. The most famous cashierless store in the world needed a thousand people watching video.
It stands.
And here's the thing I keep coming back to. Those thousand people are invisible by design. They're not in the store. They're not in the marketing. They're not in the earnings call. And when the system gets shut down, they're the first ones gone.
They're the most disposable part of the most celebrated part of the operation.
Which is a strange thing to build on purpose.
It's not strange if you accept the premise that the labor is supposed to be temporary. The whole business case assumes the humans are a bridge to the day the model handles it alone. So the humans are structurally temporary. The company isn't hiding that from itself.
It's hiding it from everybody else.
It's hiding it from everybody else.
So how far has the illusion gone? Give me the honest answer.
Further than the examples suggest, because the examples are the ones that got caught. And the catching usually happens in one of two ways. Either a journalist gets a worker on the record, or a company has to disclose something to investors and the disclosure contradicts the marketing.
So the ones we know about are the ones with a paper trail.
The ones we know about are the ones with a paper trail. And there's a whole category we can't see, which is products where the human involvement is disclosed in the terms of service in paragraph forty and nobody reads it.
Which is technically honest.
Technically honest and functionally deceptive. There's a real difference between a disclosure and a disclosure that a human being would ever encounter.
How would a customer even check?
They mostly can't. You can't audit a black box from the outside. You can time the responses. You can try to trip it up. But a hybrid system is designed to look seamless, and seamlessness is the whole product.
So the trust problem isn't that companies lie. It's that the truth is unfalsifiable from where the customer sits.
Which is why the interesting cases keep coming from insiders. The worker who talks. The contractor who posts. The employee who leaves and writes it down.
And the incentive to fake it is only going up, because the valuation premium for the word AI is enormous and the cost of hiding a human is a payroll line nobody audits.
There's a term for the rebranding too. AI washing. Taking a service that was always human powered and putting AI on the label.
Is that always a lie?
No, and this is where I want to be careful. Sometimes there's a real model in the pipeline doing real work and the human is the exception handler. That's a hybrid, and hybrids are legitimate. The problem is when the marketing describes the exception handler's job as the model's job.
So the question isn't is there AI in here. The question is who does the work when it matters.
Who does the work when it matters. That's the test I'd give anybody.
Hilbert.
Hilbert: The room had no windows and the air conditioning was set to sixty-four and you could not change it.
Okay.
Hilbert: Twenty terminals. We were told we were the AI's backup. That was the phrase in the onboarding. The AI's backup. The AI was called something with a Z in it. I don't remember which letter they put after the Z.
So there was a real model.
Hilbert: There was a real model and it was terrible. That's the part your episode is missing. It wasn't that they had no AI. They had one. It answered about one in five messages correctly. The rest it either ignored or it made something up about a return policy that did not exist.
You were covering for it.
Hilbert: I was covering for it. My job was to watch the queue and when the model stalled, I typed the reply. And the reply went out under the model's name. The customer saw a little icon and a name and they thought they were talking to the thing.
How did you know when to take over?
Hilbert: There was a timer. If the model hadn't answered in forty seconds, a box came up on my screen. And the box said take over. And I took over.
Forty seconds.
Hilbert: Forty seconds. And if you answered faster than that, you got a note from the supervisor, because it looked suspicious. So you had to sit there and watch the clock run down before you were allowed to help somebody.
You had to perform slowness.
Hilbert: You had to perform slowness. And you had to perform it in a particular way. No contractions. That was a rule. Do not use contractions. And no exclamation points. And no saying sorry more than once, because the model didn't apologize twice.
They were coaching you to sound like a machine.
Hilbert: They were coaching me to sound like a machine that was pretending to be a person. Which is a narrow target. I got written up once for saying I'm instead of I am.
What was the dashboard like?
Hilbert: It had a button. Top right corner. It said AI mode.
It said AI mode.
Hilbert: It said AI mode. And when you pressed it, your replies got a little more formal. It swapped some words. It put a period at the end of things that didn't need one. That was it. That was the whole feature.
The button that made you sound like the AI was a thesaurus.
Hilbert: It was a thesaurus with a font change.
And the customers?
Hilbert: The customers thanked the AI. All the time. Thank you for being so patient. You're the most helpful chatbot I've ever used. I had a woman tell me I was better than the last one. I was the last one. I was also the one before that.
How long did you last?
Hilbert: Three weeks. I quit on a Thursday. The supervisor asked me why and I said I couldn't stand pretending to be a machine, and she said everybody says that in the first month.
What happened to the company?
Hilbert: I don't know. I got a letter about a class action about four years later. I never filled it out. My brother in law says I should have. He's in the trade. He's not to be trusted on anything, but he says everybody in that building was doing the same thing and nobody ever told the customers.
He's probably right about that part.
Hilbert: He's right about that part. Anyway. I've got to go let somebody in. I'm the only one with the key.
The button was called AI mode.
The button was called AI mode and it made you sound more formal.
That's the whole industry. A button labeled AI mode that just changes the tone.
A thousand people in a room pressing it.
Here's what I'd leave people with. The next time you're talking to something that claims to be AI, the honest question isn't whether it's smart. It's who's on the other end and whether anybody told you.
The uncomfortable answer is that for a lot of products, the human is doing the work and the model is doing the introduction. The introduction is the product.
Which is why the disclosure matters more than the technology. The technology is often fine. The representation is the liability.
You're going to say that sentence a lot.
I'm going to say it once and let it sit.
Fair. There's one thing from the research that didn't make the main thread and it's worth thirty seconds. The workers on these platforms often can't tell you what they worked on, because they signed agreements that prevent them from saying which client they labeled for. So the invisibility isn't just that nobody sees them. It's that they're contractually unable to be seen.
The supply chain is not just hidden. It's sealed.
Sealed by the same paperwork that keeps the customer from knowing.
That's a good place to leave it. The open question is how many products right now are doing exactly this and we simply don't have the insider yet.
Whether there's an honest version. Transparent human assisted AI, labeled as such, sold as such. I think there is. I don't think anybody's buying it.
Thanks to Hilbert Flumingtop, our producer, who is currently letting somebody in somewhere.
This has been My Weird Prompts.
If you got something out of this one, leave us a review. It helps people find the show.
We'll be back soon.
