Extracting Data From a Blurry Phone Photo of a Receipt

The complaint is always the same: the model read the merchant name and the date perfectly and got the total wrong. That is not bad luck, and it is not something a better prompt fixes. It is a direct consequence of what a language model can and cannot bring to a damaged image.

Blur destroys the amounts first

A vision model reading a smeared word is doing two things at once: recognising glyphs, and choosing the most likely word given every other word around it. When the glyphs are ambiguous, the second job carries the first. A blurred rendering of “SUBTOTAL” is still recoverable from four legible letters, because almost nothing else appears in that position on a receipt.

Now look at 16.89. There is no prior. Every digit is independent of every other digit, every substitution produces a perfectly valid amount, and the substitutions blur makes easy are the worst ones: 8 and 3 differ by a single stroke, 6 and 5 differ by whether one curve closes, 1 and 7 differ by a serif, 0 and 8 differ by a waist. So the model is doing pure glyph recognition on exactly the field that matters, with no second job to fall back on, and it is doing it fluently — it will return a confident, well-formed, plausible, wrong number.

This is the whole shape of the problem, and it decides the strategy. You are not trying to make the model read better. You are trying to find out which of the numbers it returned is wrong, and you have to do that with something other than the model’s own confidence.

Three different failures look identical

“Blurry” covers three distinct degradations with three different prospects, and it is worth knowing which one you have before you spend anything on it.

• Motion blur. The phone moved during exposure. The smear is directional — strokes perpendicular to the motion survive and strokes parallel to it merge. It is often partly recoverable, and it is the one case where a second frame from the same burst is usually sharp.
• Defocus. The lens locked onto the table, not the paper. The smear is isotropic and thin strokes vanish evenly. Small print — which on a receipt is the line items — goes first, while the large print at the top survives, which is why these photos look better than they are.
• Compression mush. The image was resized and re-encoded on its way through three apps. This is not blur; it is 8×8 block artefacts that a sharpening pass will amplify into strokes that were never on the paper. Check the file size and the dimensions before assuming an optics problem.

The practical consequence: if the photo was taken seconds ago and the person is still standing there, asking for another photo beats everything below. Nothing in image processing competes with more photons.

What to do to the image, in order

Order matters, because several of these steps destroy the input for the ones after them.

1. Keep the native resolution. Vision models tile large images and charge by tile, which tempts people to downscale first. Downscaling a blurry receipt removes the marginal pixels that separate a 3 from an 8. Crop instead of scaling.
2. De-warp before anything else. A receipt photographed at an angle has a scale gradient down the strip and often a curl, so the bottom third is smaller and more compressed than the top. Correcting perspective from the paper’s four corners makes the glyph height uniform, which every later step assumes.
3. Grayscale, then a local threshold. Receipts are unevenly lit — a phone flash puts a hot spot in the middle and shadow at the edges. A global threshold picks one cut point for the whole sheet and erases whichever half it was not tuned for. An adaptive threshold computed over a small window keeps both.
4. Do not sharpen aggressively, and do not upscale generatively. Unsharp masking on a defocused image raises ringing into strokes. A diffusion-based upscaler is worse: it will produce a crisp, beautiful, invented digit, and you will have no way to tell it apart from a real one. A blank is recoverable; a confident fabrication is not.
5. Re-ask on a tight crop. Send the amounts column on its own as a second request. Because the model tiles, a narrow crop spends far more of its effective resolution on the glyphs you care about. This is the single cheapest quality gain available and it costs one extra call.

The receipt checks itself

A receipt is a small system of equations. The line items sum to the subtotal, tax is a rate applied to some or all of the subtotal, and the printed total is the subtotal plus tax plus any gratuity. So you can test an extraction against itself with no ground truth at all:

items      = [12.00, 4.50, 16.89, 3.25]
subtotal   = 36.64      # printed
tax        = 3.11       # printed
total      = 39.75      # printed

sum(items) - subtotal = 0.00     -> items agree with subtotal
subtotal + tax - total = 0.00    -> footer agrees with itself
tax / subtotal         = 0.0849  -> plausible sales tax rate

The useful part is what happens when it fails, because the residual names the error. A residual that exactly equals one of the extracted line items means a row was read twice or dropped. A residual of exactly 5.00 against a four-item receipt points at a single digit in the units column — a 1 read as a 6, or a 6 read as a 1. A residual that is a multiple of 0.90 in the tenths place is the classic 8-for-3 or 3-for-8 substitution. And a residual that is exactly twice a line item is a sign error, usually a discount or a returned item printed in parentheses and parsed as positive.

None of this needs the model. It is arithmetic over the numbers you already have, it runs in microseconds, and it tells you which field to re-crop and re-ask about instead of re-running the whole page. When the residual is small and localised, a targeted second pass on that one row usually closes it.

Two cautions on the identity itself. Tax is frequently not a flat rate over the whole subtotal — groceries and prepared food are taxed differently in the same basket, so a tax check that fails by a few cents may be correct rather than broken. And a receipt with a handwritten gratuity will never foot against its printed total; that is a different problem, covered in extracting tip and gratuity amounts.

What to return when it is unreadable

The failure mode worth designing against is a pipeline that always returns a complete object. Make every amount field nullable and pair every null with a reason drawn from a small closed set — illegible, cropped_out, occluded, contradicts_total — so downstream code can tell “the receipt had no tip” apart from “the tip line was a smear”. Those two are the same JSON if your only representation of failure is a zero.

Decide in advance which fields are load-bearing. For most expense workflows the total, the date and the merchant are required and the line items are a nice-to-have, which means a photo that yields a footing total and three unreadable rows is a partial success, not a rejection. Route on that: full pass, partial with the missing fields named, or human review. The general machinery for scoring and queuing that decision lives in per-field extraction confidence and does not need repeating here.