Benchmark: doc2json on messy real-world documents

On phone photos the plain prompt also made up 11.8 values per 100 fields, against 0.0 for doc2json. It missed 73.1% of their line items, against 3.8% for doc2json.

Totals for all 50 documents, including the 5 old scans we score but do not publish (see the caveats):

• system doc2json or plain_prompt
• fields_scored, fields_correct header fields in the answer key, and how many matched
• made_up_values, made_up_checks made-up values, and how many returned values were checked for them
• items_in_key, items_matched, items_extra line items in the answer key, matched, and added with no match
• wrong_fields the header fields that did not match, separated by semicolons

• Speed. The plain prompt is faster: a median of 4.7 seconds per document against 7.6 seconds for doc2json on the 45 published documents (4.2 s against 7.5 s across all 50). If speed matters more than accuracy on messy files, that is a real trade-off.
• Clean multi-page PDFs. Both scored 100% on the synthetic bank statements and long invoices (70 of 70 fields each).
• Valid JSON. Both returned valid, schema-shaped JSON on every document. Valid JSON is not the same as correct values.

• Documents. 50 documents. Public datasets: 15 CORD phone-photo receipts, 10 SROIE scanned receipts, 10 Mendeley sample invoices and 5 RVL-CDIP old office scans (invoice class). We added 5 long synthetic invoices and 5 multi-page bank statements we generated, where the generator input is the answer key. 4 images were rotated for the rotated-page test (2 CORD receipts and 2 Mendeley invoices, by 90 or 180 degrees).
• Published results. 45 of the 50 documents come from sources that allow published results. The 5 RVL-CDIP scans are scored but not published, so every number on this page uses the 45, except the all-50 totals, which are labelled.
• Answer keys. Built from the dataset labels and completed by hand, then frozen before either system ran. A person spot-checked them against the original images. A field that is not printed on the document is recorded as empty, which is how made-up values are caught.
• Same schema for both. One JSON schema per document type (receipt, invoice, statement), sent unchanged to both systems.
• Baseline. A small general-purpose LLM (GPT-5 nano, minimal reasoning) with one plain prompt: extract these fields and return only JSON matching this schema. JSON mode on, no examples, no retries for quality, no pre-processing. This is roughly what a developer would wire up in an afternoon.
• One run each. Each document went through each system once. Only transport failures with no output (rate limits, server errors, timeouts) could be retried.

• Made-up value: a value returned for a field that is empty on the document, or a wrong number or date that appears nowhere on the document.
• Line item missed: an item in the answer key with no matching item in the output.
• Extra line item: an item in the output with no match in the answer key.
• Valid JSON: the output parses and validates against the schema, where any single value may be null.

• Small sample. 50 documents is enough to show a pattern, not to give precise rates. A difference of a few points on one slice can come from one or two documents, and with one run each, a single field can flip between runs.
• Old scans are not published. We also score 5 low-resolution invoice scans from RVL-CDIP, but we do not publish their per-document results or images. They only appear in the all-50 totals. On one of them doc2json read every description but shifted the amounts by one row, so 14 of the 16 line items it missed across all 50 documents come from that one scan. Very old, low-resolution scans are still the hardest case.
• Print-and-photograph test not done yet. We plan to print some documents and photograph them at an angle. That variant is not in these results.
• One baseline. We tested one small model with one plain prompt. A larger model, or a tuned prompt with retries and validation, would likely score differently. We have not tested those.
• Synthetic documents. The invoices and statements we generated are cleaner and more regular than many real ones.

• CORD. Consolidated Receipt Dataset, NAVER Clova. Licensed CC BY 4.0. Source
• SROIE. ICDAR 2019 Robust Reading Challenge on Scanned Receipts OCR and Information Extraction. Licensed CC BY 4.0. Source
• Samples of electronic invoices. Mendeley Data, doi 10.17632/tnj49gpmtz.2. Licensed CC BY 4.0. We used the copy published on Hugging Face as katanaml-org/invoices-donut-data-v1. Source
• RVL-CDIP. A subset of the IIT-CDIP collection, drawn from the Legacy Tobacco Documents Library (tobacco industry litigation archive). Scored internally, not shown. Source
• Changes made. Some images were rotated (2 CORD receipts and 2 Mendeley invoices) and re-saved. Labels were normalized and completed by hand for the benchmark schemas.