Receipt scanning with a language model: what works, and how to catch an invented number

A photographed receipt is the worst input there is. Creased, shot at an angle, half faded thermal print, the name in Hebrew and the amounts in Latin digits, and sometimes it is a screenshot of a payment app rather than a receipt at all. Every bookkeeping system still wants to turn that into a structured row with a supplier, a date, an amount and VAT.

A vision-capable language model does this far better than classic OCR, and fails differently. Wrong OCR produces gibberish that is easy to spot. A wrong model produces a number that looks entirely reasonable. Everything else here is designed around that difference.

What has to come out of a receipt

Before picking a model, write the field list, because it determines how hard the task is: supplier name, tax ID, document date, document number, document type (tax invoice, receipt, combined tax invoice and receipt, proforma), amount before VAT, VAT amount, total due, currency, and payment method.

Three of those are genuinely hard. Document type, because the heading is not always printed and is sometimes printed wrong by whoever issued it. The tax ID, because it appears in ten formats and often sits next to a phone number in the same size. And the split between amounts, because some receipts show only a total and leave VAT to be derived.

Classic OCR versus a vision model

Approach	What you get	Weakness
Tesseract	Raw text, free, local	Collapses on angled shots and on Hebrew in a non-standard face
Managed OCR service	Text with coordinates, reasonably reliable	You still write the logic deciding which number is VAT
Vision model	Structured fields directly	Invents a value when unsure, unless you stop it
Both together	OCR text as extra input to the model	More infrastructure, but gives you an anchor to cross-check

What a vision model gives you and OCR does not is layout understanding. It knows the number in the bottom corner, under the word "total", is the total, even when the table is not aligned. That logic is exactly what gets hand-written in every OCR project and never quite works.

Running both is worth considering when accuracy is critical: run OCR separately and compare the numbers the model returned against the text read off the image. A number that appears nowhere in the OCR output is an excellent candidate for human review.

Schema-forced output

The first mistake everyone makes is asking for "return JSON" and then parsing the reply. Structured output solves this at the API level: you define a schema, and the response is validated against it.

import Anthropic from '@anthropic-ai/sdk';
import { z } from 'zod';
import { zodOutputFormat } from '@anthropic-ai/sdk/helpers/zod';
import { readFileSync } from 'node:fs';

const Receipt = z.object({
  supplier_name: z.string().nullable(),
  supplier_tax_id: z.string().nullable(),
  document_type: z.enum(['tax_invoice', 'receipt', 'tax_invoice_receipt', 'proforma', 'unknown']),
  document_number: z.string().nullable(),
  document_date: z.string().nullable(),
  currency: z.string().nullable(),
  subtotal: z.number().nullable(),
  vat_amount: z.number().nullable(),
  total: z.number().nullable(),
  notes: z.string().nullable(),
});

const client = new Anthropic();

const response = await client.messages.parse({
  model: 'claude-opus-5',
  max_tokens: 4096,
  system: SYSTEM_PROMPT,
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'image',
          source: {
            type: 'base64',
            media_type: 'image/jpeg',
            data: readFileSync('receipt.jpg').toString('base64'),
          },
        },
        { type: 'text', text: 'Extract the fields from this receipt.' },
      ],
    },
  ],
  output_config: { format: zodOutputFormat(Receipt) },
});

const receipt = response.parsed_output;

Two schema decisions matter more than the prompt.

Every numeric field is a number, not a string. If the model returns "1,240.00", the schema fails and you learn about it immediately, instead of discovering after parseFloat that the amount became 1.

Every field is nullable. That sounds minor, and it is the difference between a system that admits it cannot see the tax ID and one that guesses it. A model with no way to return null will fill something in.

The prompt that works

The prompt is short. A few rules earn their place:

Copy numbers as printed, do not compute. If the receipt shows only a total, subtotal and vat_amount stay null. Arithmetic is the code's job, because in code it is always right.

Return null for any field not clearly present, and do not guess. Say explicitly that an empty field beats a guess.

Dates as YYYY-MM-DD. An Israeli receipt prints 03/09/26, and without an explicit instruction that the order is day-month-year, a model trained mostly on American text will hesitate.

document_type comes from the list only. That is why it is an enum in the schema rather than free text.

Describe anything unusual in notes: a cropped image, blurred text, two documents in one shot. That field is read by a person, and it produces more value than any attempt to extract a numeric confidence score.

How to catch an invented number

This is where a demo separates from a product. The model does not give you a real per-field probability, and asking it to "rate your confidence from 1 to 10" produces a number that looks like confidence and behaves like noise. Instead, check the output against the world.

The first check is arithmetic. When all three amounts are present, subtotal + vat_amount = total must hold, within a rounding cent. One fabricated field almost always breaks that equality.

The second is the VAT rate. vat_amount / subtotal should come out at 18% (the Israeli rate since 2025), or 0 for an exempt or export transaction. A result of 13% says one of the two numbers is wrong, without yet telling you which.

The third is a check digit. Israeli tax IDs and company numbers follow the same check-digit algorithm, and ten lines of code reject a large share of digit-level misreads.

const validTaxId = (id) => {
  const digits = String(id).padStart(9, '0');
  if (!/^\d{9}$/.test(digits)) return false;
  const sum = [...digits].reduce((acc, char, i) => {
    const step = Number(char) * ((i % 2) + 1);
    return acc + (step > 9 ? step - 9 : step);
  }, 0);
  return sum % 10 === 0;
};

The fourth is date sanity: not in the future, and not before the business opened. A receipt dated 2019 scanned today is usually a misread final digit.

The fifth is duplication. Same supplier, same document number, same amount, already in the system. That catches both a double upload and a misread that collided with an existing row.

Every failed check sends the document to a manual review queue with the suspect field flagged. That is a real confidence measure, computed from the data rather than requested from the model.

An eval before any prompt change

Once something is in production, every prompt change is a risk, and "it seems better" is not a measurement. You need a fixed set of real receipts with hand-recorded correct answers. Two hundred documents is enough to start, if they are varied: faded thermal, an angled shot, a clean PDF from a large supplier, a screenshot, and a receipt in a foreign currency.

Measure per field, not per document. "87% of documents correct" hides what matters: maybe total is right 99% of the time and supplier_tax_id only 60%, and that is a completely different change to your workload. Measure how many documents drop to manual review too, because a prompt that returns more null looks more accurate while doing less work.

Run the set before every prompt or model change, and keep the results. The batch API costs half and fits this exactly, since nobody is waiting on the answer.

What it costs and how long it takes

An image is consumed as tokens according to its size, so sending a 12-megapixel photo is waste: the model downscales an oversized image anyway, and all you bought was a longer upload. Resize on your side, keep the text legible, and check the real cost by counting tokens instead of estimating.

const count = await client.messages.countTokens({
  model: 'claude-opus-5',
  messages,
});

Beyond that, three levers. A sensibly sized image instead of the original; a fixed system prompt with prompt caching, since it is identical on every request; and batch for anything that does not need an immediate answer.

Model choice is a decision to measure, not to guess. A smaller, cheaper model may handle clean receipts and fail precisely on the faded thermal one, which is the case where a mistake is expensive. The only way to know is to run both over the same set and compare per-field accuracy against cost per thousand documents.

What leaves your system

A receipt carries a supplier name, amounts, sometimes a customer name and a partial card number. Sending it to an external API is a transfer of data to a third party, and that belongs in your privacy policy and data processing agreement, not only in internal documentation.

Three questions worth answering before launch rather than after: what the provider stores and for how long, whether you keep the original image for audit and for how long, and what happens when a customer asks to delete their account. If the answer to the last one does not include the files in object storage, it is not an answer.

The draft stays a draft

The model's output is a proposal. It lands in a pre-filled form with the image open beside the fields, and a person approves or corrects it. A field that failed a check is flagged. A field that came back null is simply empty, which beats a wrong number nobody will check.

The corrections users make are the asset accumulating here. Every correction is a free labelled example, and a receipt someone fixed by hand is worth more to your eval set than anything you invented.

Share