From meal photo to FHIR

The feature sounds simple: photograph your lunch, get a nutrition log. The engineering problem is that every step of that sentence is probabilistic — vision models guess, food databases disagree, and health data has to end up in a format clinicians and other systems can trust. This is the pipeline that makes it dependable.

The model call (Claude Sonnet 4.5 on Amazon Bedrock) is forced to return JSON matching a strict schema — ingredient names, quantities, units, confidence. Free-text answers are never parsed with regexes and hope; a response that fails validation is retried, and one that fails twice degrades to a “needs review” state instead of writing garbage into a health record. LLM output is treated like any other untrusted user input.

The end of the pipeline isn’t a bespoke meals table — it’s a FHIR resource in Medplum. That one decision buys interoperability with the rest of the healthcare stack: care teams, analytics, and any future integration read the same standard resource, and access control comes with the platform instead of being reinvented.

A full pipeline run takes seconds — too long for a spinner. Results stream to the client over SSE as they’re produced, so the app shows ingredients appearing while macros and scoring are still in flight. Same total latency, entirely different feel.

The scoring pass at the end is deliberately boring: deterministic rules for allergen safety and plan fit, applied to the structured data — not another model call. When a number decides whether a meal is flagged for an allergy, you want arithmetic, not vibes.