Cover art for Image Annotation When Experts Are the Bottleneck
Episode 6611 custom

Image Annotation When Experts Are the Bottleneck

The WikiChurches art historian annotated just 631 prototypical bounding boxes, narrowing 117 style classes down to four.

21:56 Transcript

0:00
0:00
Episode Details
Episode ID
MWP-6814
Published
Duration
21:56
Audio
Direct link
Pipeline
V5.32
TTS Engine
omnivoice
Script Writing Agent
GLM-5.3

AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.

The prompt

The prompt that started this episode, as sent in (production notes for the hosts removed).

Image annotation tools, with a specific use case in mind: my wife Hannah is an architect, and something she'd find really interesting is training a model for architectural classification. That's a very different training task from the audio and text annotation I've been doing. You're zooming into large, detailed images, drawing bounding boxes or polygons around features, and applying classes that need expert judgement. What are the annotation environments people use for this kind of image work, both cloud and self-hostable, and what UI features make the difference when a domain expert is labelling thousands of images?

Annotating architectural styles for machine learning is far more complex than labelling audio or text. Drawing on the NeurIPS WikiChurches paper, architectural classification hinges on tiny visual details, like a forty-pixel-wide fleuron, alongside non-exclusive, hierarchical labels where buildings span multiple eras. When a domain expert's time is the scarce resource, tooling choices and interface design matter. Self-hostable platforms like CVAT, Label Studio, Labelme, and Supervisely offer AI-assisted features such as point-to-polygon masking and text-to-annotation, though each carries setup quirks like browser restrictions or security defaults that block local backends. Rather than forcing an expert to label thousands of ambiguous images, the most effective workflow curates prototypical examples, pairs bounding boxes with free-text reasoning, and prioritizes fast zooming and category-first browsing.

Sources

What the research for this episode read before the script was written. Primary sources first.

  1. CVAT repo, README and releases v2.81.0 primary v2.83.0 ; retrieved 2026-11-15; last push 2026-11-15; latest release v2.83.0 (2026-11-12).
  2. Label Studio repo, README and releases 1.23.1/1.23.2/nightly ; retrieved 2026-11-15; last push 2026-11-14; stable 1.23.2 (2026-09-29). primary
  3. Labelme repo, README and releases v7.9.1 primary v7.11.0 ; retrieved 2026-11-15; last push 2026-11-14; latest v7.11.0 (2026-11-14).
  4. Supervisely SDK repo, README and releases v6.74.54 primary v6.74.56 ; retrieved 2026-11-15; last push 2026-11-12.
  5. Barz & Denzler, WikiChurches: A Fine-Grained Dataset of Architectural Styles with Real-World Challenges, NeurIPS 2021 Datasets & Benchmarks Track, v2 revised 2021-10-18; full text read (pp. 1 primary 5) ; retrieved 2026-11-15.
  6. CVAT Manual Annotation docs, v2.83.0 ; retrieved 2026-11-15 (page returned mostly navigation; feature detail taken from the repo README and release notes instead). primary
  7. Zhong et al., ArchiLense: A Framework for Quantitative Analysis of Architectural Styles Based on Vision Large Language Models, v3 revised 2025-08-02; retrieved 2026-11-15.
  8. Kubba et al., Multimodal Cultural Heritage Architectural Style Classification for Residential Buildings in the UAE Based on CLIP Embeddings and SVM, submitted 2026-09-15 (SITA 2025); retrieved 2026-11-15.
  9. Wei, Li & Shamir, Computer Analysis of Architecture Using Automatic Image Understanding, J. Data Mining & Digital Humanities, v4 revised 2020-07-09; retrieved 2026-11-15.
  10. Shrestha et al., FEVA: Fast Event Video Annotation Tool, v2 revised 2023-01-07; retrieved 2026-11-15.
  11. Banerjee et al., SWAN Enabling Fast and Mobile Histopathology Image Annotation through Swipeable Interfaces, submitted 2025-11-11; retrieved 2026-11-15.
  12. Längkvist et al., Interactive user interface based on Convolutional Auto-encoders for annotating CT-scans, submitted 2019-04-26; retrieved 2026-11-15.
  13. Schomaker, A large-scale field test on word-image classification in large historical document collections, submitted 2019-04-17; retrieved 2026-11-15.
  14. Image Annotation Tools Compared: CVAT, Label Studio, Roboflow, V7 and Potato; retrieved 2026-11-15.
  15. 12 Best Image Annotation Tools in 2026: CVAT, Roboflow & More; pricing as displayed August 18, 2026; retrieved 2026-11-15.
  16. Best Computer Vision Annotation Tools 2026; retrieved 2026-11-15.
  17. Ultralytics, Best Image Annotation Platforms for Vision Teams; retrieved 2026-11-15.

Mentions

  • CLIP Vision-language model used for open-vocab detection
  • ControlNet Conditioning control for diffusion image models
  • CVAT Annotation platform with team and video support
  • label studio Open source data labeling tool
  • Labelme Lightweight desktop image annotation app
  • Roboflow Annotate Hosted end-to-end annotation service
  • Segment Anything Meta's click-to-mask segmentation model
  • Supervisely Enterprise annotation platform with live training
  • WikiChurches NeurIPS dataset of expert-annotated church images
  • Wikidata Open knowledge base used for collection queries

Downloads

Episode Audio

Download the full episode as an MP3 file

Download MP3
Transcript (TXT)

Plain text transcript file

Episode Book (PDF)

The episode's record — date, duration, models, sources — with the full transcript

#6611: Image Annotation When Experts Are the Bottleneck

Corn
Daniel has spent months annotating audio and text, and now Hannah wants in on the training-data game. She's an architect, and the project is a model that classifies architectural styles. His question: what annotation environments do people actually use for detailed image work, cloud and self-hostable, and what UI features matter when a domain expert is labelling thousands of images. The part I like is that last bit. When the annotator is the expert, the tool isn't just catching keystrokes. It's handling the scarce resource.
Herman
And it's a different job from what he's been doing. I keep coming back to the audio case, because he knows it. You place a boundary in time on a one-dimensional signal, and once you've heard the thing, there's usually no argument about where it starts and stops. Text is the same, just highlighted instead of cut. Architecture gives you none of that comfort.
Corn
None.
Herman
There's a paper called WikiChurches, from NeurIPS, which is literally this task. Nine and a half thousand images of churches, expert bounding boxes, architectural style classification. They say determining a building's style is hard even for experts, because the categorization depends on a combination of subtle and sometimes tiny visual features, and they name quatrefoils and fleurons. A fleuron is a carved flower at the top of a finial. It can be forty pixels wide in a photo of an entire facade.
Corn
Forty pixels. The whole label decision can hang on something the size of a thumbnail on a five-megapixel image.
Herman
That's the first difference. The second is that the labels aren't mutually exclusive. Roskilde Cathedral in Denmark is Brick Gothic in the catalogue, but parts of it were rebuilt across centuries, so sections read as Renaissance, Neoclassicism, Byzantine Revival, even Modern. The paper concludes that style recognition is actually a multi-label problem. And the third difference: the labels are hierarchical, and people are inconsistent about depth. Thousands of buildings labelled simply Gothic, only eighty-one labelled at the finest level of the hierarchy, because most people stop at the coarse label.
Corn
So in audio, the tool needs a waveform and a pair of buttons. Here, the tool has to support zooming into a multi-megapixel image, judgement calls about whether a feature is diagnostic, and a decision about which level of a taxonomy to commit to, with no objective answer. That's what Hannah's signing up for.
Herman
Which is why the choice of environment actually matters here more than it did for his audio work. Let's do the landscape first, then the differences, then the interface features, then the workflow.
Corn
Go.
Herman
Self-hostable side, four names keep coming up. CVAT is the reference computer-vision annotator. MIT licence, seventeen thousand stars, runs with a docker compose up, supports boxes, polygons, masks, keypoints, even 3D point clouds. Twenty-plus export formats, COCO, YOLO, Pascal VOC, so the model side is never a problem. Python SDK, REST API.
Corn
The catches.
Herman
Two, and they're specific. The good interactors, click a point and get a mask from Segment Anything, are in the free version, but the newer SAM two and three auto-labelling and the AI agents are paid Enterprise only. And it officially supports Chromium browsers only, Safari is not supported. So check your browser before you set Hannah up.
Corn
I like that the free version's price is a browser.
Herman
Label Studio is the multimodal one, Apache licence, twenty-eight thousand stars. It does audio, text, images, video, time series. This matters for Daniel specifically, because one self-hosted instance could hold his audio spans and Hannah's facade boxes side by side. Installs with pip or docker.
Corn
You have a but face.
Herman
I have a big but. The September release turned security defaults on, and one of them blocks ML backends on localhost and private networks. Which is exactly where a self-hoster's backend runs. You have to know the flag exists and deliberately disable it. Otherwise you spend an afternoon wondering why your model-assisted labelling is silent.
Corn
A security default whose practical effect is to break the thing you installed the software for. Herman, you must have hit that one personally.
Herman
I hit it within an hour, yes. Clean install, backend up, no suggestions arriving. I assumed it was my firewall, my ports, my karma. It was a default.
Corn
Labelme is the third one, right? The desktop one.
Herman
Labelme, yes. Python and Qt, GPL, a lightweight desktop application rather than a server. Polygons, rectangles, points, plus something I find interesting. It does point-to-polygon with SAM, so you click inside a window tracery and get the outline. And it does text-to-annotation with newer models, so you can literally type what you want outlined.
Corn
Type quatrefoil, receive geometry.
Herman
Roughly. And in this month's release it added AI-Find All, which finds every object matching a text prompt or one example box across a batch, and you refine it with counter-examples while holding Shift. Draw one rose window, get every rose window proposed, then draw over the mistakes. For an architect who knows exactly what she's looking for, that could collapse a week of work into an afternoon of reviewing.
Corn
Then it's doing the finding and she's doing the confirming.
Herman
Which is the right division of labour for the scarce resource. Fourth, Supervisely, enterprise-grade, Apache SDK, handles images, video, 3D, medical imaging. Its interesting feature is live training sessions, you start one mid-project, pick your classes, seed it with the annotations you've already made, and the model starts suggesting while you label. The model improves during the labelling instead of after it.
Corn
So those are the environments. The self-hosted ones are free, they run on his hardware, and each has a landmine with his name on it.
Herman
Cloud side, briefly. Roboflow Annotate is the strongest hosted end-to-end option, but it's cloud-only, no true self-hosting on the free or pro tiers, and on the free Public plan your data is visible in Roboflow Universe. Hannah's photographs of client buildings, in a public gallery, in exchange for the free tier.
Corn
That's the actual price. Not the zero dollars.
Herman
Right. Hosted CVAT exists, third-party listings put it around thirty-three dollars per user per month, Enterprise from twelve thousand a year, but those are roundups, not the vendor, so treat them as approximate. Label Studio paid starts around ninety-nine a month, Supervisely Pro around a hundred and ninety-nine euros. The real cost of self-hosting, though, isn't the licence, it's the ownership. Someone has to handle upgrades, access control, backups, performance, and the integrations that move data into training. That's Daniel, or it's nobody.
Corn
He'll do it. He'll enjoy it. The landmine question is whether Hannah enjoys it.
Herman
She won't touch it, and she shouldn't have to.
Corn
Okay, that's the landscape. Now the differences, because I want to push on this. The interesting claim in that paper isn't that images are harder than audio. It's where the difficulty sits.
Herman
It sits in three places. Tiny features, which we covered. Non-exclusive labels. And uneven label precision. The last one is sneaky, because it's not a tooling problem at all, it's a data problem wearing a tooling costume. If your classes come from a catalogue like Wikidata, you inherit every inconsistency in the catalogue. The paper found about six and a half percent label noise, and thirty-one of the sixty-four level-one styles had exactly one image each.
Corn
One image. A class with a population of one.
Herman
One church, one style label, one photo. That's not a class, that's an anecdote. And this is where it connects to a failure mode I've read about in other domains, a historical-document field test that found deep learning accuracy with more than a thousand one-hot output classes drops to nearly zero, while an older bag-of-visual-words approach held around eighty-seven percent.
Corn
Wait. The paper's own dataset has a hundred and seventeen style labels. If Hannah builds a full taxonomy, she walks straight into that cliff.
Herman
She does if she tries to use the whole taxonomy. The WikiChurches team didn't. For the expert annotation stage they deliberately narrowed to four well-studied styles, Romanesque, Gothic, Renaissance, Baroque, which together cover about eighty-nine percent of the images anyway. They had the courage to throw away a hundred and thirteen classes.
Corn
So the expert wasn't annotating the dataset. The expert was annotating a curated subset of it.
Herman
And here's the number that should set expectations for Hannah. All of that expert knowledge, an art historian, given free rein, produced six hundred and thirty-one bounding boxes across a hundred and thirty-nine churches. Not labels for nine and a half thousand images. Six hundred and thirty-one boxes. Weeks of expert time, deliberately spent on the prototypical cases only. The protocol literally asked for churches that were, quote, pure in style and prototypical for that period.
Corn
Skip everything that argues with you.
Herman
Ambiguous buildings were skipped by design. That's not laziness, it's curation.
Corn
Interesting. My instinct would have been to push the expert through the hard cases, because that's where the model is weakest. But this says the opposite, spend the expert where the signal is cleanest.
Herman
There's a second reading too, which I like even more. The six hundred and thirty-one boxes each carry a free-text description of the feature inside, written by the art historian, and boxes could be grouped when several elements form one structure. So the expert's actual reasoning, what she noticed and why it mattered, is sitting in the data. There's a recent paper on Emirati residential architecture that leans all the way into that, they embedded the expert's textual descriptions alongside the images with CLIP and got ninety-eight percent accuracy across eight style clusters. The words aren't decoration. They're a training input.
Corn
Which answers a question Daniel didn't ask. The expert's descriptions aren't metadata around the labels. They might be the labels.
Herman
That's the transition I wanted. If the expert is the bottleneck, and she is, she's the only person in the building who can tell a fleuron from a finial, then the interface features that support expert judgement aren't nice-to-haves. They're the whole game. So let's do those.
Corn
Before we do, we have our producer's segment.
Hilbert
Hilbert's Mid-Show Haiku. The Badge Is Forty Pixels Wide. One box drawn at four hundred percent. The expert squints. The label waits.
Herman
The badge is forty pixels wide. So was the fleuron.
Corn
So the interface features. Zoom first, because everything in that protocol depended on it.
Herman
The WikiChurches expert drew boxes in a popup window, by dragging the mouse over a larger version of the image. A bespoke interface built for one art historian, and the enlargement is treated as part of the annotation design, not a viewer convenience. Any tool Hannah uses has to make zooming a one-gesture, keyboard-first thing, not a menu.
Corn
And my claw situation says something about that, incidentally. Pinch-to-zoom on a trackpad was designed for fingers. Give me keys and a wheel or give me death.
Herman
Second feature, category-first browsing. The interface let the expert browse all images per category, including sub-categories. Read that again. The taxonomy is the navigation. She moves through the collection by style, the thing she knows, not through a flat folder of nine thousand JPEGs, the thing the file system knows.
Corn
That's backwards from how every annotation tool defaults. They're built around the image queue, because the annotator is assumed to be interchangeable. The moment the annotator is the expert, the queue is the wrong shape.
Herman
Third, the AI-assisted drawing. We touched it with Labelme, but it belongs in the list. SAM as an interactor, click a point, get a mask, is in CVAT's free tier too. The expert confirms rather than traces. Fourth, example-based finding, the AI-Find All pattern, find every rose window from one example, corrected with counter-examples. Fifth, quality control that understands geometry, and this is more interesting than it sounds. CVAT's October release added a variable overlap threshold depending on the area of the ground-truth box.
Corn
Because a forty-pixel fleuron and a whole facade can't be judged by the same tolerance.
Herman
A one-pixel miss on a fleuron is a catastrophic error in relative terms, and invisible on a facade. A single fixed threshold would call one of them perfect and the other wrong. Sixth, consensus. CVAT added replica jobs, multiple annotators on the same job. Which matters because of a finding from medical annotation I keep quoting, six radiologists segmenting CT scans came out with a kappa of about zero point six nine, and the study's conclusion was that the inter-reader variation is itself a challenge for the definition of ground truth. The disagreement among experts isn't noise in the data. It's data.
Corn
Though Hannah is a practice of one. Who's her second radiologist?
Herman
She has Daniel, and it's useful even if he's wrong in a consistent direction, because then you learn which judgements are his. But if she can't get a second expert, the honest answer is her ground truth is an opinion with excellent credentials.
Corn
Seventh, seventh. Model in the loop.
Herman
Seventh, model in the loop, the Supervisely live-training pattern, seed it with her first hundred boxes and let it suggest. And there's evidence people behave well under this. That CT study found users who leaned on the suggestions, prioritizing learning speed over model accuracy in their words, gave the tool higher usability scores. Nobody is honoured for tracing every outline by hand.
Corn
Which is a cousin of the skip lesson. Let the human spend herself only where she's needed.
Herman
Eighth, and this is the one I'd put money on. Interaction count. There was a survey of fifty-nine video annotation tools, and its finding was blunt, most have overwhelming interfaces, poor interaction techniques, and difficult-to-understand features, leading to longer annotation time, label inconsistencies, and user fatigue. The survey's own redesign cut required interactions by thirty-six percent against five popular tools. Thirty-six percent, just from fewer clicks per decision.
Corn
So the misconception is pick the tool with the most features. The truth is closer to pick the tool with the fewest clicks.
Herman
For an expert, yes. A crowdsourced annotator needs guardrails and checkboxes. Hannah needs speed, zoom, and a skip button.
Hilbert
The cast-iron storefronts. That was the winter of eighty-five, a small downtown, and I photographed every building for the historical society and marked the ironwork on paper prints, in pencil, because that's what they'd asked for and the pencil came out of my own pocket.
Corn
On paper prints.
Hilbert
Photographic prints, Herman, letter size, and I drew on them with a chinagraph pencil, which doesn't smudge when the print sweats in a cold building. The marking was the easy part. You'd think the catalogue was the buildings we kept. It wasn't. The work was the shoebox. Every building I decided was not interesting enough, I printed anyway and put it in a separate box, labelled REJECTED in the same chinagraph, and when I finished the winter I had more in that box than in the catalogue.
Herman
How many more?
Hilbert
I didn't count them then and I won't now. By ninety-two, four of the rejected ones had been demolished, and the shoebox was the only photograph anyone had. I still have it. Corn, I bring one photograph from it to family gatherings and ask if anyone remembers the building, and mostly nobody does, and I put it back.
Corn
You're running a small mercy mission for buildings nobody fought for.
Hilbert
I'm running nothing. I put it back in the box.
Corn
He's right, though, that the rejects are a record. Half the judgement in annotation is what you refuse to label.
Herman
That's the part the tools don't store. The skip is a click that goes nowhere. Imagine if every skipped image carried the reason, and that's, no, I'm just going to say it plainly. The skip log is the expert's negative space, and no environment I know of keeps it.
Corn
So, the workflow end to end, because Daniel asked what it actually looks like.
Herman
WikiChurches is the honest template, and it's less glamorous than people expect. Collect, and their collection was a query against Wikidata, ten thousand churches. Filter, they cut about three percent interiors with a pretrained classifier and then checked it by hand. Resize, smaller side to a maximum of about twelve hundred and eighty pixels. Then the expert stage, and that's the six hundred and thirty-one boxes we talked about, on a hundred and thirty-nine hand-picked churches, four styles, free-text on every box.
Corn
Then the model, which by your historical-document caution can't be a hundred and seventeen classes.
Herman
It can't. The bench isn't huge either. There's a recent benchmark of about seventeen hundred architectural images with a vision-language framework that reports ninety-two percent consistency with expert annotations, and the CLIP approach with the eight style clusters at ninety-eight. Small, well-defined class sets, expert text as a signal, part-level boxes for the small diagnostic features. That's the whole recipe.
Corn
We did an episode once on architectural AI with ControlNet and ComfyUI, on the generation side of that world, and I remember the same asymmetry. Everyone wants to talk about the model. Nobody wants to talk about the curation.
Herman
And curation is where Hannah's actual skill lives. So here's the one thing. If you take one thing from this episode, let it be this: when the annotator is the expert, the problem isn't how to get more labels, it's how to spend her attention. Four styles, not a hundred and seventeen. Prototypical buildings only. Free-text on every box. Six hundred and thirty-one boxes from an art historian was a complete, publishable dataset.
Corn
Narrow the problem until the expert's judgement fits in it.
Herman
And the open question this leaves me with, the thing I'd like to see built, is an annotation environment organised around the expert's taxonomy rather than the model's output format. Hannah's label tree as the front page, her skip decisions logged, her free-text treated as signal rather than decoration. The tools are all pointed at the model. The interesting one points at the architect.
Corn
Our producer is Hilbert Flumingtop, who keeps a shoebox we will never see. If you enjoyed this one, try episode twenty, Architectural AI; episode twenty-one ninety-six, The Invisible Workforce Behind AI; and episode eight, Building Your Own Whisper. This has been My Weird Prompts. If the show earns a corner of your week, a review helps other people find us.
Herman
We'll be back soon. See you tomorrow.

This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error