Skip to content

Google Gemini (AI / OCR) ​

PPP uses AI models for three features:

  • Medical sticker OCR — extracting UR number, date of birth, and gender from photographed patient stickers in the surgical logbook
  • Operating list OCR — extracting surgical case details from photographs of hospital theatre lists (#3107)
  • Reflection quality scoring — grading the quality of a CPD reflection and suggesting improvements
  • PDP generation — turning a user's CPD history into a draft Personal Development Plan

Historical note — "Gemini" is a misnomer today

PPP originally used Google Gemini for all three features. The main PPP backend image and text services have since been migrated to Azure OpenAI (gpt-4o-mini as the primary model, gpt-4o as the fallback) for availability and regional-data-residency reasons. The environment variable is still called GEMINI_API_KEY for continuity, and the WhatsApp ops bot (api/whatsapp-bot/) still uses the real Gemini API. This page covers both paths so readers don't get confused when they see Gemini references in secret lineage and Azure OpenAI references in the service code.

1. Where it's wired ​

Main PPP backend (Azure OpenAI today) ​

FilePurpose
api/backend-ppp/src/services/imageService/index.tsAzure OpenAI vision client, medical sticker + operating list OCR
api/backend-ppp/src/services/reflectionQualityService/index.tsReflection quality scoring; AI text provider is Azure OpenAI
api/backend-ppp/src/services/pdpGenerationService/index.tsPDP generation; AzureOpenAITextProvider as the default
api/backend-ppp/src/routers/procedures/analyzeImage/tRPC procedure called by the mobile/web client

WhatsApp ops bot (real Gemini API) ​

FilePurpose
api/whatsapp-bot/src/services/gemini-parser.tsClassifies incoming WhatsApp messages into GitHub issues

The WhatsApp bot is a separate Azure Function App. It ingests support messages, asks Gemini to classify them, and then creates a GitHub issue with the right labels. It is intentionally decoupled from the main PPP backend. The Pulumi infrastructure project (api/infra-whatsapp-bot/) was removed in #5124 — the bot's Azure resources were never deployed.

2. Medical sticker OCR ​

What it does ​

A clinician on the ward photographs a patient's sticker with the PPP mobile app. The image is uploaded to blob storage via a presigned URL, the tRPC client calls analyzeImage.medicalSticker, and the backend:

  1. Validates the MIME type is one of image/jpeg | png | webp | heic | heif
  2. Loads the editable prompt from the system_prompts table (key: medical-sticker-ocr) with a hard-coded fallback
  3. Sends the image + prompt to Azure OpenAI (gpt-4o-mini primary, gpt-4o fallback)
  4. Parses the JSON response into { urn, dob, gender, confidence }
  5. Returns the extracted fields to the client, which pre-fills the logbook form

The frontend displays each extracted field with an indicator of whether it was AI-populated so the clinician can review before saving.

What it does not do ​

  • No patient identifying data is retained on disk beyond the blob already uploaded to PPP storage. The response is streamed back to the client and discarded.
  • No cross-referencing — the OCR does not call any external PAS or EHR system. The extracted UR number is stored as a plain string in the logbook; it is not resolved to anything.
  • No free-text output — the prompt strictly enforces a JSON shape, and the server rejects responses that do not parse.

Prompt management ​

The prompt lives in system_prompts and can be edited via the Super Admin UI. The DEFAULT_MEDICAL_STICKER_PROMPT constant in imageService/index.ts is a safety fallback for when the DB-backed prompt is unavailable. If you edit the DB prompt, make sure it still enforces the JSON shape — the parser depends on it.

3. Operating list OCR (#3107) ​

Similar shape to the sticker OCR, but the prompt extracts an array of surgical cases from a photographed theatre list. Each case has patient name, UR number, procedure, surgeon, role, time slot, hospital, and notes. The list also surfaces listDate and hospitalName at the top level.

Prompt key: operating-list-ocr (DB-backed, with DEFAULT_OPERATING_LIST_PROMPT as fallback). Same Azure OpenAI model chain. The feature is still behind a rollout flag in UAT at the time of writing.

4. Reflection quality scoring and PDP generation ​

Both features use an AiTextProvider abstraction in api/backend-ppp/src/services/pdpGenerationService/index.ts:

ts
const aiProvider = provider ?? new AzureOpenAITextProvider();

The provider interface is deliberately narrow so tests can inject mocks without spinning up a real model. In production the default is Azure OpenAI; in tests the provider is always mocked.

  • Reflection quality scoring — takes a user's reflection text and returns a score + structured feedback. Used to help trainees improve their CPD reflections over time.
  • PDP generation — takes a user's recent CPD activities and returns a draft Personal Development Plan structured by the RCS PDP template. The user reviews and edits the draft before saving.

5. Configuration ​

Main PPP backend env vars (Azure OpenAI) ​

VariablePurposeRequired
AZURE_OPENAI_API_KEYAzure OpenAI API key from the configured resourceYes
AZURE_OPENAI_ENDPOINTAzure OpenAI resource endpoint (https://<name>.openai.azure.com)Yes
AZURE_OPENAI_API_VERSIONPreview API version (defaults to 2024-08-01-preview)No

The client code rejects placeholder values (REPLACE_ME, your-api-key-here, empty string, etc.) at request time and returns a tRPC PRECONDITION_FAILED error so a misconfigured deploy fails loudly instead of silently.

Legacy / WhatsApp bot env var (real Gemini) ​

VariablePurposeRequired for
GEMINI_API_KEYGoogle AI Studio API key for generativelanguage.googleapis.comWhatsApp bot

This variable is still listed in docs/site/development/backend/environment.md as "required for OCR image analysis (Gemini)", which dates from before the Azure OpenAI migration. It remains injected into the PPP Container App by Pulumi for backward compatibility and for any code path that falls back to the direct Gemini API, but the current main image and text services do not consume it.

Secret lineage ​

Both AZURE_OPENAI_API_KEY and GEMINI_API_KEY flow through the same pipeline as every other external credential:

  1. Source of truth: Azure Key Vault (AZURE-OPENAI-API-KEY, GEMINI-API-KEY)
  2. Exported into Doppler for developer access (ppp/uat, ppp/prod)
  3. Read by Pulumi (api/infra-azure/src/index.ts) via requireEnvSecret at deploy time
  4. Injected into the Container App env at provisioning time

See api/infra-azure/SECRET_LINEAGE.md for the full table.

6. Resilience ​

Image analysis calls go through the shared visionCircuitBreaker:

  • Circuit breaker — trips after 5 consecutive failures, resets after 60 seconds
  • Fallback model — gpt-4o-mini is tried first; on failure the service retries with gpt-4o before giving up
  • Timeout — handled by the Azure OpenAI SDK's built-in client

When the breaker is open, analyzeImage returns a tRPC SERVICE_UNAVAILABLE error with a user-friendly message ("OCR is temporarily unavailable, please enter the fields manually"). The signup and logbook flows all degrade gracefully — the AI path is an accelerator, never a requirement.

7. Failure modes ​

FailureUser impactOperator action
Azure OpenAI rate limitsOCR calls 429; circuit breaker opens after 5 failuresReview request volume; request a quota increase
AZURE_OPENAI_API_KEY missing or placeholderanalyzeImage returns PRECONDITION_FAILED immediatelyRe-apply Pulumi; check Doppler binding
Model returns non-JSONHandler logs and returns a generic extraction-failed errorReview the system prompt for regression
Primary model unavailableService retries with the fallback model transparentlyNone — self-heals
Image too large (>20MB)Pre-check fails with a clear error; client shows messageNone — enforced by MAX_IMAGE_FILE_SIZE_BYTES
Unsupported MIMEPre-check fails; client shows "unsupported image format"None — frontend already filters to supported types

8. Cost and billing ​

Azure OpenAI ​

PPP is billed for Azure OpenAI on a per-token basis with separate rates for input and output tokens. For the image path:

  • gpt-4o-mini — the primary model, chosen specifically because it is roughly an order of magnitude cheaper than gpt-4o per million tokens while still producing accurate results for the structured OCR prompts PPP uses.
  • gpt-4o — fallback only. Only used when gpt-4o-mini fails, which should be rare.

Cost observability for Azure OpenAI is part of the same Azure cost tooling as the rest of the infrastructure. There is no dedicated Grafana dashboard yet — cost is tracked in the monthly Azure invoice review.

Sticker OCR is the hot path: one sticker per surgical case logged, typically 0.5–2 kB of output. Operating list OCR is more expensive (larger images, longer structured JSON outputs) but lower volume (one list covers 5–20 cases).

Real Gemini (WhatsApp bot) ​

The WhatsApp bot uses gemini-1.5-flash at low volume (a few hundred classifications per month). Cost is bounded by the fact that each message is ≤1 kB and the model is the cheapest tier. Billed directly to the Google Cloud project tied to GEMINI_API_KEY.

9. Security and privacy ​

  • No prompt logging — prompts and model responses are not written to application logs. Diagnostic logs contain the service name, user ID, action, and error code only.
  • Placeholder detection — the image service refuses to call Azure OpenAI if the API key looks like a placeholder (REPLACE_ME, your-api-key-here, etc.), preventing accidental calls from dev machines that never got their .env set up.
  • Input validation — image MIME types are validated both in the tRPC Zod schema and at the service layer. The prompt-injection surface is small because the user supplies an image, not free text.
  • Retention — raw images are stored in PPP blob storage under the existing retention policy. Extracted fields are stored in the users / logbook tables as plain strings. Nothing is retained inside Azure OpenAI by PPP.

10. Testing ​

bash
cd api/backend-ppp
pnpm test -- imageService
pnpm test -- pdpGenerationService
pnpm test -- reflectionQualityService

The test suite mocks the Azure OpenAI client using handlers in api/backend-ppp/src/__fixtures__/handlers/ (still named gemini.ts for historical reasons — see handlers directory for the current set) and covers:

  • Successful OCR extraction (all fields populated)
  • Partial extraction (some fields missing)
  • Unreadable image (all fields null)
  • Model timeout and retry
  • Circuit breaker behaviour
  • Unsupported MIME rejection
  • Prompt loading from DB with fallback to hard-coded default

Contract tests in api/backend-ppp/src/__fixtures__/external-services.contracts.test.ts exercise the full external-service integration path with recorded fixtures.

Internal documentation — org members only