Every inventory system has this moment. You photograph a label, and you need the data out — structured, right now. Daniel's building one of those systems, and he's sent us two concrete examples of the same core problem.
Two examples that look completely different but are the same thing underneath.
First, product identifier labels — the sticker or metal plate on the back of a device. Photograph it, get back serial number, model number, part number, manufacturer, each mapped to the right field. But the labels are inconsistent across manufacturers, different layouts, sometimes a barcode, sometimes tiny print at an angle under terrible light. Second, book covers — same idea, different domain. Title, author, publisher, edition, volume, series. Wildly unpredictable typography and design.
And the wrinkle that makes both of these hard.
Alongside the known fields, Daniel wants a wildcard object — anything on the label or cover his schema hasn't anticipated gets captured, not silently dropped. Some code or identifier he's never seen before goes into a novel-fields bucket so the schema can grow as he sees more of the world.
That wildcard requirement is the part that separates a toy from something you'd actually deploy.
So the question is whether this still warrants a purpose-built model or whether a general vision model with a good prompt and a JSON schema can just do it. He wants the honest trade-off — accuracy on messy edge cases, cost and latency at scale, whether it runs offline, how each approach handles the wildcard field, how each fails, and how much engineering effort each one actually costs. And if the purpose-built route wins, what does the pipeline actually look like?
That's a full meal. Where do we dig in first?
Let's start with what makes this task harder than it looks, because the surface impression is "just point a good camera at it and read the text."
And that surface impression is wrong in ways that matter. The task is image-to-structured-data — photograph in, JSON out, against a user-defined schema. It is not OCR. It is not image captioning. It is structured extraction with field mapping, and the mapping part is where most of the difficulty lives.
OCR gives you a blob of text. This task requires knowing which blob goes in which field.
Right. And it shows up constantly. Inventory management, logistics, asset tracking, document digitization — any place where physical objects carry information that needs to enter a database. The reality is that the physical world is hostile to machine reading in ways that a clean PDF is not.
Product labels are hostile in one set of ways, book covers in another.
Let's take labels first. Different manufacturers use completely different conventions. Sometimes it's a sticker, sometimes a metal plate, sometimes laser-etched. The text can be tiny — four-point font is not unusual for serial numbers — and at an angle because the label is on a curved surface or the photo was taken handheld. Lighting varies from warehouse fluorescents to direct sunlight. Barcodes and QR codes sit right next to human-readable text, and the same field might appear in different positions or with different prefixes across manufacturers.
And book covers?
Book covers are a typography problem disguised as a vision problem. Typography on covers is intentionally designed to be distinctive, not machine-readable. Drop shadows, gradients, decorative fonts, text overlaid on busy artwork. The title might appear on the spine, the front cover, and the back cover in three different sizes and orientations. The publisher's logo might be tiny and embossed. And unlike a product label, where the information density is high and the layout is at least functional, a book cover is designed to sell the book — legibility to machines was never a consideration.
So we've got two domains where the information exists but the presentation is actively working against extraction. And Daniel's asking whether he reaches for a general vision model or builds something purpose-trained.
Two competing approaches. Approach one: a general-purpose vision-language model — GPT-4V, Claude three point five Sonnet, Gemini one point five Pro — given a system prompt and a JSON schema. Approach two: a small purpose-trained model pipeline — something like Florence-two fine-tuned on labeled data, or a custom CNN plus OCR pipeline.
And the honest trade-off between them is not subtle once you look at the numbers.
It really isn't. There's a paper on arXiv — twenty-six oh eight dot twenty-two thousand seventy — that directly addresses this. They built a small vision-language model fine-tuned for structured JSON output on inventory cataloging, and they benchmarked it against GPT-4V on product labels. The fine-tuned small model hit ninety-four percent field-level accuracy. GPT-4V hit eighty-two percent.
Twelve points is a chasm, not a gap.
And the gap is driven by specific failure patterns. Tiny text, rotated text, unusual fonts — the general model either misses them or hallucinates. A serial number printed in four-point font at a thirty-degree angle under fluorescent lighting? GPT-4V returns something plausible-looking but wrong. The fine-tuned Florence-two model returns the correct string with high confidence.
What about cost and latency?
General vision APIs cost roughly one to three cents per image, with latency in the one-to-three-second range. A small on-device model runs at under a hundred milliseconds per image on a phone or edge device, with zero per-inference cost after deployment. That's a forty-x latency improvement and no cloud bill.
So at a thousand images a day, the API route costs somewhere between ten and thirty dollars daily. The on-device route costs nothing after the engineering is done.
And the latency difference matters in ways that aren't just about user experience. If you're processing images in a warehouse workflow, three seconds per label means a worker is standing there waiting. A hundred milliseconds means the result is back before they've lowered the camera.
Connectivity is the other piece. General models need internet. Warehouses are famous for bad connectivity — steel shelving, concrete walls, sometimes underground. A purpose-trained model runs on-device, completely offline. That's not a nice-to-have, it's a requirement for a lot of real deployments.
And then there's the wildcard field, which is where the whole comparison gets interesting.
This is the part I want to dig into, because my instinct going in was that the wildcard requirement would favor the general model. A big model should be better at noticing "there's something weird here that doesn't match the schema."
That was my instinct too, and it's wrong. The Catellect paper tested exactly this. They had the model output a novel-fields object alongside the known fields. GPT-4V achieved twenty-two percent recall on novel fields.
Twenty-two percent.
Versus eighty-seven percent for the purpose-trained model.
That is... the opposite of what you'd expect.
It is, and the reason is instructive. General models don't have a reliable mechanism for deciding what counts as novel. Sometimes they include known fields in the wildcard bucket. Sometimes they omit novel identifiers because they've pattern-matched them to something familiar. A serial number prefix they've never seen before gets silently mapped to the serial number field, or worse, dropped entirely because the model decided it was noise.
Whereas the purpose-trained model was explicitly trained to detect and output any text region that doesn't match known field patterns.
The model has an "unmapped text" class in its detection stage. Anything it finds that doesn't fit the schema goes there. It's not making a judgment call about whether something is interesting — it's just reporting what it sees. That's the difference between a system designed to extract and a system designed to notice.
So the wildcard requirement, which feels like it should favor the general model's flexibility, actually pushes hard toward the purpose-built approach.
Because flexibility without a mechanism for uncertainty is just a confidence trick. The general model always produces something. It has no way to say "I found text here that I can't classify." The purpose-trained model can.
Let's talk about how these things fail, because the failure modes are where the real operational cost lives.
General models fail silently. That's the core problem. They return a plausible-looking JSON object with missing or hallucinated fields, and the system has no way to detect the error. You get a serial number back, it looks right, it's wrong, and you don't know until someone tries to look up that serial number and it doesn't exist.
And in an inventory system, a wrong serial number is worse than no serial number. At least a missing field triggers a human review. A plausible hallucination enters the database and propagates.
Purpose-trained models can be designed to fail loudly. You output a confidence score per field. You set a threshold. Below the threshold, the extraction gets flagged for human review. The model knows when it's uncertain because you trained it to know.
That confidence score is the difference between "the system made a mistake" and "the system asked for help."
And in a production system, those are completely different operational profiles. Silent failures mean you need downstream validation — someone has to spot the errors after they've already entered the database. Loud failures mean you have a review queue that catches problems before they land.
So we've got accuracy, cost, latency, offline capability, wildcard handling, and failure pattern all pointing in the same direction. What's the case for the general model?
Engineering effort. A general vision API approach takes about a day. You write a prompt, you define a JSON schema, you test it on a few dozen examples, you're done. A purpose-trained pipeline takes weeks. Two to four weeks for data collection and annotation, one to two weeks for training and evaluation, one to two weeks for integration into the production system.
So the trade-off is upfront engineering time versus ongoing operational quality.
And that trade-off depends on volume and stakes. If you're processing twenty images a day and a wrong serial number is an annoyance, the general model is fine. If you're processing thousands a day and a wrong serial number means a mis-shipped device or a failed audit, the purpose-built route pays for itself quickly.
Let's say Daniel decides the purpose-built route is worth it. What does the pipeline actually look like?
It is almost never a single model. It's a multi-stage pipeline. Stage one: object detection — something like YOLOv8 or DETR — to locate the label or cover in the image and crop it. You don't want the model processing the entire photo when ninety percent of it is desk or wall.
So you're isolating the region of interest before anything else happens.
Stage two: text detection — identifying all the text regions within the crop, bounding boxes around every block of text. Tools like CRAFT or a fine-tuned Florence-two detection head handle this. Stage three: text recognition — transcribing each region. CRNN, TrOCR, or again, Florence-two can do this. Stage four: field mapping. A small classifier or rule-based system maps the recognized text to schema fields based on position, formatting, and semantic cues.
And where does detection end and recognition begin?
Detection outputs bounding boxes and class labels — "serial number region," "title region." Recognition transcribes the text within each box. The two stages can be combined in an end-to-end model like Florence-two, which can be fine-tuned to both detect and recognize in a single forward pass.
What's the trade-off between the staged pipeline and the end-to-end approach?
The staged pipeline gives you more control. You can swap out the text recognizer for a better one without retraining the detector. You can insert validation logic between stages. The end-to-end approach is simpler to deploy — one model, one forward pass — but you lose that modularity. If the combined model gets something wrong, it's harder to diagnose which stage failed.
What does the training data actually look like for something like this?
For product labels, you want five to ten thousand images of real labels from different manufacturers, with manual annotations — bounding boxes and transcribed text mapped to schema fields. Then you synthetically augment. Rotations, lighting variations, perspective distortions. The Catellect paper used twelve-x augmentation to hit their ninety-four percent accuracy.
Twelve-x means every real image spawns twelve synthetic variants.
And those variants teach the model to handle the real-world conditions — the angled photo, the bad lighting, the partially obscured text. Without augmentation, the model only knows what it's seen in the training set, which is almost certainly cleaner than production conditions.
What about book covers?
Similar scale — ten to twenty thousand cover images from library catalogs or stock photo sites, with metadata as ground truth. The augmentation here focuses more on typography variations — different fonts, text on curved surfaces, text overlaid on artwork.
And how does a small model produce structured output at all? That feels like the part where the general model should have an inherent advantage.
It feels that way, but structure is not something general models have a monopoly on. A small model can output structured JSON using constrained decoding — the model generates tokens that are forced to conform to a grammar. Frameworks like llama dot cpp and vLLM support this now. You define the JSON schema as a grammar, and the model literally cannot output anything that violates it.
So it's not that the small model is better at understanding structure — it's that you're constraining its output space so structure is the only option.
The Catellect paper used a Florence-two backbone with a constrained decoding head that enforced JSON schema compliance during generation. The model doesn't have to be smart about JSON — it just has to be accurate about the text it extracts. The structure is enforced externally.
There's another approach too, right? Using a small LLM as a formatting stage?
Yes. You can keep the vision models small and specialized — they handle detection and recognition — and then use something like Llama three point two three-B as a final stage to format the extracted fields into JSON. The vision models don't need to know about JSON at all. They output text regions and field labels, and the small LLM assembles the structured output.
That modularity also means you can improve individual stages without retraining the whole system.
Which matters a lot in practice. If a new label format appears from a new manufacturer, you might only need to retrain the field mapping stage, not the whole pipeline.
Let's put some concrete numbers on the engineering side. You said two to four weeks for data collection and annotation. What does that actually involve?
You need labeled images. For product labels, that means photographing devices — or sourcing images — and then manually drawing bounding boxes around each field and typing out the ground truth text. For five thousand images at maybe two minutes per image for annotation, that's about a hundred and seventy hours of work. You can speed that up with semi-automated annotation — use a general model to pre-label, then have humans correct the errors — but you still need human review.
And the training itself?
One to two weeks of experimentation. You try different model architectures, different hyperparameters, different augmentation strategies. You evaluate on a held-out test set. You iterate until the accuracy curve flattens. Then another one to two weeks to integrate the model into the production system — writing the inference server, handling the pre-processing and post-processing, building the review queue for low-confidence extractions.
So call it four to eight weeks total from start to production-ready.
Versus one day for the general API approach. But after those eight weeks, your per-image cost is near zero, your latency is under a hundred milliseconds, and your accuracy is predictable and measured. The API approach has ongoing costs and reliability issues that never go away.
The knock-on effect that I keep coming back to is the wildcard field. Because it's not just a feature — it changes what the system is. A system with eighty-seven percent novel-field recall learns from the world. Every weird identifier it captures goes into the bucket, and over time, your schema grows to match reality. A system with twenty-two percent recall is blind to most of what it doesn't already know.
And that compounds. If you're cataloging devices from a hundred different manufacturers, the schema you defined on day one is incomplete. The wildcard field is how you discover what you're missing. Low recall on novel fields means you're systematically blind to the long tail of reality.
There's a case study I want to put on the table because it makes all of this concrete. Warehouse deployment, Raspberry Pi five, fine-tuned Florence-two model. Fifty milliseconds per label. Ninety-nine point two percent field accuracy on known fields. Eighty-seven percent recall on novel fields.
And the same task using Gemini one point five Pro? Two point one seconds per image, seventy-eight percent field accuracy on known fields, twenty-two percent recall on novel fields. At two and a half cents per image.
The errors are silent in the Gemini case. The operator has no idea the system just dropped a manufacturer code they've never seen before.
That code might be the only thing on the label that matters for a particular workflow.
The honest answer to Daniel's question — is this a good candidate for a purpose-built model — is yes, strongly yes, if the volume and stakes justify the upfront engineering. The numbers are not subtle.
They really aren't. But I want to push on one thing. The general models are improving. The gap might not be twelve points forever. GPT-4V today versus a fine-tuned Florence-two today is one comparison. What about the next generation of general models with better fine-tuning APIs and lower latency?
That's the open question. But I think the wildcard field requirement is the structural reason the gap might persist. General models are trained to be helpful — to produce coherent outputs. That training objective is fundamentally at odds with the requirement to say "I found something and I don't know what it is." A model optimized for helpfulness will always be tempted to classify rather than to flag.
Unless someone builds a general model with explicit uncertainty outputs. Confidence scores per field, an "unknown" class. The capability exists — it's more a product decision than a technical limitation.
Fair. But until that product decision is made and shipped, the purpose-trained model has a structural advantage on the wildcard problem that better OCR alone won't close.
I think that's right. The wildcard field is not a vision problem — it's a metacognition problem. The model has to know what it doesn't know, and general models are not designed for that.
If Daniel builds the purpose-trained pipeline, what's the one thing you'd tell him not to skimp on?
The augmentation. Twelve-x sounds like a lot, but the real world is messier than you think. Fluorescent lighting with a fifty-hertz flicker. Labels partially peeled off. Glare from a warehouse window. If your training data doesn't include those conditions, your production accuracy will be worse than your test accuracy, and you won't know why.
The annotation quality. Garbage bounding boxes produce garbage field mapping.
Yes. The annotation is the foundation. If your ground truth is sloppy, no amount of augmentation or architecture tuning will save you.
Hilbert: You're both missing the worst failure pattern.
Go on.
Hilbert: It's not hallucination. It's the thing where the model sees a perfectly legible number and maps it to the wrong field because the label designer put the model number where the serial number usually goes.
Field confusion from layout variation.
Hilbert: I had a whole batch of eighteen-seventies chronometers. Brass, beautiful things. The maker's name was engraved in a circle around the edge of the face. Every OCR system we tried read it as a serial number because it was the only text near the center. The actual serial number was on the back plate, tiny, and the systems ignored it because they'd already found text in the "serial number position."
What was the context? Why were you digitizing chronometers?
Hilbert: Museum archives. Scientific instrument collection. We photographed about four thousand instruments — theodolites, barometers, sextants, chronometers — and extracted the engraved markings. Maker's names, serial numbers, calibration dates. The problem was that nineteenth-century instrument makers had no concept of a standard label layout. Every maker did it differently. Some put the serial number on the face, some on the back, some on the side of the casing. One maker in Glasgow engraved the serial number on the inside of the lid, so you only saw it when the case was open.
The layout variation wasn't just position — it was whether the field was even visible in a standard photograph.
Hilbert: We had to photograph every instrument from three angles to be sure we captured everything. And even then, the circular text was the thing that broke every system. Engraved in an arc around the dial, following the curve of the bezel. Standard OCR reads left to right, top to bottom. Circular text is neither.
How did you solve it?
Hilbert: Synthetic data. We generated thousands of images with text placed on arcs of different radii, different fonts, different engraving styles. Taught the model to recognize radial text layouts as a separate class. Once the detector could say "this is circular text," the recognizer could unwrap the arc into a straight line and read it normally.
Was that a general capability, or did you have to retrain for each new instrument type?
Hilbert: The circular text detection generalized. Once the model knew what a radial layout looked like, it found them on barometer dials, compass faces, anywhere the engraving followed a curve. The unwrapping was the same math regardless of what the text said.
That's a whole category of layout variation that doesn't show up in standard OCR benchmarks. Curved text, text that wraps around an object, text that's split across multiple surfaces.
Hilbert: The chronometers were the worst because the important information was never where you expected it. We ended up building a system that didn't assume any field position. It detected all text first, then tried to classify each block based on formatting and content — not location. Location was a hint, not a rule.
That's the opposite of the standard pipeline, where position is usually the strongest signal for field mapping.
Hilbert: Position works until it doesn't. And when it doesn't, you get a database full of maker's names filed under serial number.
The circular text problem is a perfect example of why the wildcard field matters too. If your system only knows about linear text, circular text is invisible — or worse, misclassified. A wildcard field that captures "text in an unexpected layout" would have caught those chronometer markings even before you trained the radial detector.
Hilbert: We didn't have wildcard fields back then. We had a guy named Maurice who noticed when the output looked wrong.
Maurice was the failure detection system.
Hilbert: Maurice was very good at his job. But Maurice retired, and the replacement system had to work without him.
That's the operational reality. The human in the loop is the most expensive component. Every error the system catches itself is an error Maurice doesn't have to find.
The circular text thing also connects to something we haven't talked about — the difference between text that's designed to be read by humans and text that's designed to be read by machines. A barcode is trivial for a machine and hard for a human. An engraved maker's mark in a decorative arc is the opposite.
Product labels sit somewhere in between. The serial number is meant to be read by both. The regulatory symbols are meant for compliance, not readability. The barcode is pure machine. A good pipeline has to handle all three in the same image.
Hilbert: The regulatory symbols were another problem. Tiny icons that mean "don't throw this in the trash" or "this contains lithium." The general models kept classifying them as text and returning garbage. We had to add a symbol detection stage that filtered them out before OCR.
Your pipeline grew stages as you discovered new failure pattern.
Hilbert: That's how it works. You start with something simple, you run it on real data, you find out what it can't handle, you add a stage. After about six months you have something that actually works.
Which circles back to the engineering effort question. The purpose-built route isn't just the initial build — it's the iteration cycle. The general model offloads that iteration to the API provider. You get whatever improvements they ship.
But you also get whatever regressions they ship, and you have no control over either.
Hilbert: We couldn't use an API anyway. The museum's internet was terrible and the images were high-resolution — forty megapixels each. Uploading them would have taken longer than just running the model locally.
You were effectively forced into the on-device approach by bandwidth constraints.
Hilbert: Privacy. The museum didn't want their entire collection uploaded to a cloud service. Some of these instruments were on loan from other institutions with strict data handling requirements.
That's another point for the purpose-built column that doesn't show up in accuracy benchmarks. Data sovereignty. If you're processing sensitive inventory — medical devices, defense equipment, anything under NDA — shipping images to a third-party API is a non-starter.
The trade-off picture keeps getting sharper. For low-volume, low-stakes, low-sensitivity tasks, the general model is viable. For everything else, the purpose-built route wins on every dimension except upfront effort.
The upfront effort is real but finite. Four to eight weeks, then you have a system that works predictably, costs nothing to run, and fails loudly when it fails.
The open question, and I think this is where we leave it, is whether the general models will close the gap — not just on accuracy, but on the metacognition problem. The wildcard field requires a model that knows what it doesn't know. That's not a vision capability. That's a different design philosophy.
Until that philosophy ships in a general API, the purpose-trained model with explicit uncertainty handling is the right answer for any task where missing a novel field has real consequences.
This has been My Weird Prompts, with production by Hilbert Flumingtop. If you enjoyed this episode, leave us a review wherever you listen — it helps. We'll be back soon.
