A Zero-Shot Decision Model Beat A Fine-Tuned Classifier
So many AI workflows start with a decision. This can be a query router or the first step in an ingestion pipeline. For a while, I have leaned on LayoutLM and its successors when I wanted to classify documents. This can be fine-tuned to predict many labels and it traditionally is very performant. It’s small + fast which is a “winning combination”.
Recently, however, sparked by Jev , a new breed of model has begun to work. They are saying “non-generative”, which is to say it doesn’t generate content but rather “makes decisions”. There are quite a few write-ups about these models, but shortly after Jev’s release Cloudflare announced an open model that supported the same interface.
So I wanted to look at new versus old approaches. On one hand I had a fine-tuned LayoutLMv3: 126M parameters, 0.5 GB of weights which required OCR text (with boundaries) and an image of the page. I ran a bake-off against Cloudflare’s clef-flash, a 9B zero-shot decision model (Qwen3.5-9B backbone). I was looking to answer two questions:
- Is clef-flash a replacement for LayoutLM (and specifically the non-open-source LayoutLMv3)?
- Do I need word-level OCR bounding boxes to train the classifier, or are the line boxes sufficient?
The short version: clef-flash running locally won hands down. But like everything, the results are less conclusive than they look.
Setting Up The Test
I scored both models on three evaluation sets. clef-flash had seen none of them.
- DOJ datasets 9-12 (159 documents, 146 in the label set). This is the target data. 100 documents were seeded-random, 58 were keyword-selected from 8,147 sampled text files to cover rare types, and one FD-302 came from a neighboring Bates range. No human-labeled slice of these datasets existed, so an LLM agent (Claude) labeled each document from its first page image, following the written type criteria, before any model ran. These labels are provisional.
- HOUSE_OVERSIGHT held-out (47 pages, 42 in set). Human-labeled, taken from LayoutLMv3’s own validation split.
- Public validation (194 pages). Tobacco3482 / RVL-CDIP pages with their human dataset labels, up to 25 per type.
LayoutLMv3 ran through the pipeline with image and OCR. clef-flash was presented a single prompt with a choice question whose criteria were the type descriptions, the page image at 1000 px plus the OCR text, and one forward pass to produce the probabilities. No prompt tuning. Everything ran locally.
The Headline Numbers
Accuracy / macro-F1 over documents whose gold label is one of the eleven types. Latency is per document, warm, batch 1.
| Predictor | DOJ 9-12 (agent labels) | HO held-out (human) | Public val (human) | p50 latency | Peak GPU memory |
|---|---|---|---|---|---|
| LayoutLMv3, legacy | 0.870 / 0.63 | 0.810 / 0.53 | 0.753 / 0.71 | 0.15 s | 0.7 GiB |
| LayoutLMv3, training geometry | 0.870 / 0.63 | 0.929 / 0.75 | 0.974 / 0.98 | 0.11 s | 0.7 GiB |
| clef-flash, image + OCR text | 0.986 / 0.96 | 0.810 / 0.58 | 0.629 / 0.62 | 0.60 s | 19.2 GiB |
| clef-flash, image only | 0.993 / 0.97 | 0.810 / 0.58 | 0.634 / 0.61 | 0.55 s | 19.6 GiB |
| clef-flash, OCR text only | 0.863 / 0.65 | 0.881 / 0.65 | 0.588 / 0.55 | 0.27 s | 19.6 GiB |
On the DOJ slice the per-type picture looked lopsided:
- clef-flash got 8 of 8 FD-302 interview reports.
- LayoutLMv3 got 0 of 8, calling them forms and resumes.
- On court filings clef-flash scored 1.00 F1 against 0.40, with LayoutLMv3 calling four of them letters.
- Interview/meeting summary is not one of LayoutLMv3’s 22 labels at all, so it could never predict it.
What This Test Reveals
- The labeler and a contestant are the same kind of model. The DOJ labels came from an LLM. clef-flash is an LLM. Shared biases inflate agreement. When an LLM labels your eval set and an LLM is in the race, you have partly measured agreement, not accuracy.
- The random sample barely separates them. The random part of the slice has 87 in-set documents, 79 of them emails. LayoutLMv3 scored 97.7 % there and clef-flash 100 %. That is about two documents, and the intervals overlap.
- The rest of the gap is partly circular. The remaining difference comes from the 58 keyword-selected rare-type documents. The agent labeled those using the same type criteria clef-flash received as its prompt.
- The human-labeled sets go the other way. On HO held-out the two tie at 0.81. On public validation clef-flash loses, 0.63 against 0.75.
- The rare classes are tiny. Each rare type has 1 to 8 DOJ examples, so one document moves a type with n=3 by a third. The macro-F1 intervals are wide: 0.76-1.00 for clef-flash, 0.45-0.68 for LayoutLMv3.
- Selection bias and possible exposure. Keyword selection over-samples documents whose text names their type (FD-302 headers, case captions). And because these releases are public, clef-flash’s backbone may have seen some of them in pretraining.
clef-flash’s weak spot is clear from public validation. On low-resolution 1990s scans it read forms as letters (11 of 25), handwritten pages as letters (11), and invoices as forms (8). Handwritten letters are ambiguous under this label set: the criteria make a mostly handwritten page “handwritten,” but the model tends to pick the page’s purpose. The DOJ material (email printouts, FBI and DOJ records, court filings) does not look like that data, which is why the target-data result still matters. It just doesn’t settle the question.
The Real Finding: A Train/Serve Skew
Look at the second row of the table again. The same LayoutLMv3 checkpoint jumps from 0.753 to 0.974 on public validation, and from 0.810 to 0.929 on HO held-out, with no retraining. The only change is how the layout boxes are scaled.
The training script capped each image at 2000 px. It then scaled the OCR boxes by 1000 divided by the capped size, but the boxes were still in the original image’s pixel coordinates. The production path scaled boxes correctly against the image they belong to.
So for every training page larger than 2000 px, the model learned magnified boxes, clamped at 1000. That covers every Tobacco3482 page (some over 100 MP) and the HOUSE_OVERSIGHT pages (about 2550×3300). In production it received correctly scaled boxes it had never seen. DOJ pages (1224×1646) fall under the cap, which is why they score the same either way.
The cost was up to 22 accuracy points on the affected pages. The checkpoint only works with the training geometry, and the fix belongs in training: scale boxes against the original size, or scale them together with the image.
One caveat cuts the other way. The HO held-out pages were the training run’s validation split, which also picked the best checkpoint, and the split was by page rather than by document. Neighboring pages of one document can sit in both train and validation. That leakage inflates LayoutLMv3’s HO numbers. clef-flash, never trained on them, is unaffected.
The general lesson: when a fine-tuned model underperforms its own validation report in production, check the preprocessing path before you blame the model.
Word Boxes: Not Needed for Classification
This was the second question, and the answer was clean.
- clef-flash doesn’t need OCR at all to classify. Image only scored 0.993 against 0.986 for image plus text. Text alone dropped to 0.863. The image carries the layout signal.
- LayoutLMv3 needs layout, but not word granularity. With every box zeroed it lost 10 accuracy points and half its macro-F1 on DOJ (0.870 to 0.767). But the line-level boxes the OCR service already stores are enough. Its errors come from types it lacks, not from box granularity.
So the decision on word-level boxes moves entirely to the features that need them, highlighting and snapping, and off the classifier’s critical path.
Confidence Matters More Than Accuracy
The pipeline sends any classification with confidence below 0.6 to human review. Swapping the model changes that review load, so I measured it on the DOJ slice.
| Predictor | Sent to review (< 0.6) | Accuracy at ≥ 0.6 | Accuracy below 0.6 | Out-of-set docs at ≥ 0.6 |
|---|---|---|---|---|
| LayoutLMv3, legacy | 4.1 % | 0.907 (13 wrong of 140) | 0 right of 6 | 6 of 13 |
| clef-flash, image + text | 6.2 % | 1.000 (0 wrong of 137) | 7 right of 9 | 2 of 13 |
The review load is about the same: 4 to 7 % either way. What gets past the threshold is very different. LayoutLMv3 let 13 wrong answers through with confidence and got every one it sent to review wrong. clef-flash let none through and was right on most of what it flagged. It is under-confident, which is the failure mode you want in a review queue.
The 13 out-of-set documents (photos, “No Images Produced” placeholders, slip sheets, fully redacted pages) exposed a gap in my own question design. clef-flash had no “other” option, so it put them in report or court_filing. At 0.6 most of those would have reached review, but not all. An explicit “other” answer was required.
What It Costs
GPU memory. clef-flash peaked at 19-20 GiB in bf16 against 0.7 GiB for LayoutLMv3. Served through SGLang it reserves about 27 GB, so little else fits beside it on a 32 GB card. OCR, embeddings and a local LLM all want the same GPU. A 9B model is also not an option for the CPU stack. A quantized version could be tested to see its effectiveness and Ollama already has one.
Throughput. Classification reads page 1 of each document, so cost scales with document count, not page count. At the transformers path’s 0.4-0.6 s per document that is roughly 11-17 GPU-hours per 100,000 documents. At SGLang’s measured ~6 requests/s with 8 in flight, about 4.6 hours.
Extra pages didn’t help. clef-flash on pages 1-3 scored 0.979 against 0.986 for page 1, and added a false positive. Averaging LayoutLMv3 over pages 1-3 hurt more (0.822 against 0.870), because later pages are often attachments. Page 1 stays the default; multi-document PDFs need splitting before classification.
Free extra signals. Asking three yes/no questions in the same pass (is this page a document start, is it privileged, is it hot) cost no measurable latency. None has gold labels yet, so they are review signals, not decisions. Privileged averaged 0.72 on DOJ prosecution memos and 0.24 on emails.
Takeaways
Separate your labeler from your contestants. Agent-assigned labels are fine for a fast indicative read. They are not ground truth when an LLM is one of the things being measured.
Audit preprocessing before replacing a model. The biggest single accuracy gain in this exercise was a box-scaling fix on the incumbent.
Measure confidence, not just accuracy. Two models with the same review rate can put very different errors in front of users.
Design for “none of the above.” Real document releases contain placeholders, slip sheets and redacted pages. A closed label set will force them into a wrong answer.
Price the swap. A full 9B model on a 32 GB card is significant.
Expect this class of models to improve. It’s only been a couple of weeks and the market will supply improved models.