Skip to content

Resilience Patterns for External Services ​

Overview ​

The PPP backend runs on AWS Lambda and communicates with several external services: Google Gemini AI, Stripe payments, Expo push notifications, and PostgreSQL. Any of these services can fail transiently due to network issues, rate limiting, or temporary outages.

Without resilience patterns, a single failing dependency can cascade through the system. For example, if the Gemini AI service is down and every medical sticker OCR request blocks for 60 seconds before timing out, the Lambda function burns through its concurrency limit, starving unrelated requests like payments and notifications. Cold starts compound the problem because newly-initialised Lambda instances immediately hit the same failing service.

Our resilience layer is zero-dependency -- it uses no external packages like opossum or cockatiel. The entire implementation lives in a single file (src/utils/resilience.ts) and consists of four composable primitives: retry with backoff, circuit breaker, timeout, and transient error detection.


Architecture ​

External calls are wrapped in a layered pattern where each layer protects against a different failure mode:

  Incoming Request
       |
       v
  +------------------+
  | Circuit Breaker  |  Fails fast when a service is known to be down.
  |                  |  Prevents wasting time and resources on doomed calls.
  +--------+---------+
           |
           v
  +------------------+
  |     Retry        |  Retries transient failures (network blips, 429s, 503s)
  |  (exp. backoff)  |  with exponential backoff and jitter.
  +--------+---------+
           |
           v
  +------------------+
  |    Timeout       |  Prevents a single slow call from blocking the Lambda
  |                  |  for its entire execution budget.
  +--------+---------+
           |
           v
  +------------------+
  | External Service |  Gemini, Stripe, Expo, PostgreSQL, etc.
  +------------------+

How the layers compose in code:

typescript
const result = await circuitBreaker.execute(() =>
  withRetry(
    () => withTimeout(signal => externalServiceCall(signal), 30_000, 'service-name'),
    { maxRetries: 2, baseDelayMs: 1_000, shouldRetry: isTransientError },
  ),
);

What each layer catches:

LayerProtects Against
Circuit BreakerSustained outages; prevents cascading failures
RetryTransient network errors, rate limits (429), 502/503
TimeoutHanging connections, unresponsive services

Utilities Reference ​

All utilities are exported from src/utils/resilience.ts.

withRetry(fn, options?) ​

Wraps an async function with automatic retries using exponential backoff.

Signature:

typescript
function withRetry<T>(fn: () => Promise<T>, options?: Partial<RetryOptions>): Promise<T>;

Options (RetryOptions):

OptionTypeDefaultDescription
maxRetriesnumber3Maximum retry attempts (total calls = maxRetries + 1)
baseDelayMsnumber1000Base delay before exponential calculation
maxDelayMsnumber30000Upper cap on any single delay
jitterbooleantrueAdd randomness to prevent thundering herd
shouldRetry(error: unknown) => booleanundefinedPredicate to decide if an error is retryable. If provided and returns false, the error is thrown immediately without further retries.

How exponential backoff + jitter works:

The delay for attempt n is calculated as:

exponential = baseDelayMs * 2^n
capped      = min(exponential, maxDelayMs)
delay       = jitter ? random(0, capped) : capped

With defaults (baseDelayMs: 1000, jitter on):

AttemptExponentialCappedActual (with jitter)
01000ms1000ms0 - 1000ms
12000ms2000ms0 - 2000ms
24000ms4000ms0 - 4000ms
38000ms8000ms0 - 8000ms

Jitter ("full jitter" strategy) randomises the delay between 0 and the capped value. This prevents multiple Lambda instances from retrying in lockstep and overwhelming the recovering service.

When to use:

  • Network calls that can fail transiently (DNS, TLS, TCP resets)
  • APIs that return 429 (rate limited) or 5xx (server error)
  • Any idempotent operation where repeating the call is safe

Example:

typescript
import { withRetry, isTransientError } from '../../utils/resilience';

const data = await withRetry(
  () => fetch('https://api.example.com/data').then(r => r.json()),
  { maxRetries: 2, baseDelayMs: 500, shouldRetry: isTransientError },
);

CircuitBreaker ​

Implements the circuit breaker state machine to prevent cascading failures when a service is down.

State Machine:

                 success
            +----------------+
            |                |
            v                |
  +--------+------+    +----+------+
  |    CLOSED     |--->| HALF_OPEN |
  | (normal flow) |    | (1 probe) |
  +-------+-------+    +-----+-----+
          |                   |
          | N failures        | failure
          v                   v
  +-------+-------+          |
  |     OPEN      |<---------+
  | (fast reject) |
  +-------+-------+
          |
          | resetTimeoutMs elapsed
          |
          +-----> HALF_OPEN (try one probe request)
  • CLOSED -- requests flow through normally. Each failure increments a counter. When the counter reaches failureThreshold, the circuit opens.
  • OPEN -- all requests are immediately rejected with CircuitBreakerOpenError. After resetTimeoutMs elapses, the circuit transitions to HALF_OPEN.
  • HALF_OPEN -- a single "probe" request is allowed through. If it succeeds, the circuit closes. If it fails, the circuit reopens.

Constructor options (CircuitBreakerOptions):

OptionTypeDescription
failureThresholdnumberConsecutive failures before the circuit opens
resetTimeoutMsnumberTime in ms before OPEN transitions to HALF_OPEN
namestringHuman-readable name for log messages

Methods:

MethodDescription
execute<T>(fn: () => Promise<T>)Runs fn through the circuit breaker
getState(): CircuitStateReturns current state: 'CLOSED', 'OPEN', or 'HALF_OPEN'
reset(): voidManually resets the circuit to CLOSED

When to use:

  • Any external service where sustained failure should cause fast rejection
  • Services that need time to recover (opening the circuit gives the service breathing room)
  • Critical paths where hanging on a dead service wastes Lambda execution time

Example:

typescript
import { CircuitBreaker, CircuitBreakerOpenError } from '../../utils/resilience';

const breaker = new CircuitBreaker({
  name: 'my-service',
  failureThreshold: 5,
  resetTimeoutMs: 30_000,
});

try {
  const result = await breaker.execute(() => callExternalService());
} catch (error) {
  if (error instanceof CircuitBreakerOpenError) {
    // Service is known to be down, return cached/fallback data
    return fallbackValue;
  }
  throw error;
}

withTimeout(fn, timeoutMs, operationName?) ​

Wraps an async operation with a hard timeout. If the operation does not resolve within timeoutMs, a TimeoutError is thrown and an AbortSignal is triggered.

Signature:

typescript
function withTimeout<T>(
  fn: (signal: AbortSignal) => Promise<T>,
  timeoutMs: number,
  operationName?: string,
): Promise<T>;

Parameters:

ParameterTypeDescription
fn(signal: AbortSignal) => Promise<T>The async operation. Receives an AbortSignal for cancellation support.
timeoutMsnumberMaximum time to wait in milliseconds
operationNamestring (optional)Label included in the TimeoutError message

AbortSignal support:

The function receives an AbortSignal that is triggered when the timeout fires. If the external SDK supports cancellation via AbortSignal (e.g., fetch), the underlying request is actually cancelled rather than just ignored.

typescript
const data = await withTimeout(
  signal => fetch('https://api.example.com/data', { signal }),
  5_000,
  'fetch-data',
);

When to use:

  • Any external call that could hang indefinitely
  • Operations where you need a predictable upper bound on execution time
  • Lambda functions with strict execution time budgets

isTransientError(error) ​

A predicate that inspects an error and returns true if it looks like a transient failure worth retrying.

What it detects:

CategoryMatches
TimeoutErrorAlways transient
Error message contenteconnreset, etimedout, econnrefused, network, socket hang up, fetch failed
Node.js error codesECONNRESET, ETIMEDOUT, ECONNREFUSED, EPIPE, EAI_AGAIN
HTTP status codes429 (rate limited), 502, 503, 504 (upstream)

When to use:

Pass it as the shouldRetry predicate to withRetry so that non-transient errors (400 Bad Request, 401 Unauthorized, 404 Not Found) fail immediately without wasting time on retries.

typescript
await withRetry(() => callApi(), { shouldRetry: isTransientError });

Custom Errors ​

CircuitBreakerOpenError ​

Thrown by CircuitBreaker.execute() when the circuit is OPEN.

PropertyTypeDescription
namestring'CircuitBreakerOpenError'
circuitNamestringThe name of the circuit breaker
messagestringHuman-readable description

TimeoutError ​

Thrown by withTimeout() when the operation exceeds the time limit.

PropertyTypeDescription
namestring'TimeoutError'
timeoutMsnumberThe timeout value that was exceeded
operationNamestring | undefinedThe label passed to withTimeout

RetryExhaustedError ​

Thrown by withRetry() when all attempts have failed.

PropertyTypeDescription
namestring'RetryExhaustedError'
attemptsnumberTotal number of attempts made
lastErrorunknownThe error from the final failed attempt

Service Integration Guide ​

Google Gemini AI ​

File: src/services/imageService/index.ts

Configuration:

typescript
const geminiCircuitBreaker = new CircuitBreaker({
  name: 'gemini-ai',
  failureThreshold: 5,
  resetTimeoutMs: 60_000, // 60 seconds
});

// Usage in analyzeMedicalSticker():
const result = await geminiCircuitBreaker.execute(() =>
  withRetry(
    () =>
      withTimeout(
        _signal => this.model.generateContent([prompt, imageData]),
        30_000, // 30 second timeout per attempt
        'gemini-generate-content',
      ),
    { maxRetries: 2, baseDelayMs: 1_000, shouldRetry: isTransientError },
  ),
);

Why these values:

  • Circuit breaker at 5 failures / 60s reset: Gemini AI is a non-critical feature (OCR is a convenience). Five failures indicate a real outage, and 60 seconds gives Google's infrastructure time to recover.
  • 2 retries with 1s base delay: AI inference is expensive; we retry twice for transient errors but do not hammer the API.
  • 30s timeout: Gemini vision analysis of medical stickers involves uploading image data and running inference. 30 seconds is generous but necessary for large images.

Fallback model:

When the primary model (gemini-2.0-flash) fails with a non-transient error, the service automatically retries with a fallback model (gemini-1.5-flash). The fallback attempt uses withTimeout only (no circuit breaker, no retry) to avoid tripping the circuit breaker further.

typescript
// Primary fails -> try fallback with timeout only
const result = await withTimeout(
  _signal => this.model.generateContent([prompt, imageData]),
  30_000,
  'gemini-fallback-generate-content',
);

Stripe Payments ​

File: src/services/paymentService/index.ts

Configuration for writes (createPaymentIntent, createUpgradePaymentIntent):

typescript
const stripeCircuitBreaker = new CircuitBreaker({
  name: 'stripe',
  failureThreshold: 5,
  resetTimeoutMs: 30_000,    // 30 seconds
});

const paymentIntent = await stripeCircuitBreaker.execute(() =>
  withRetry(
    () => stripe.paymentIntents.create({ ... }),
    { maxRetries: 2, baseDelayMs: 500, shouldRetry: isTransientError },
  ),
);

Configuration for reads (listUserPayments, attachEmbraceIdToPaymentIntent):

typescript
// No circuit breaker -- just retry
const paymentsByEmbraceId = await withRetry(
  () => stripe.paymentIntents.search({ ... }),
  { maxRetries: 2, baseDelayMs: 500, shouldRetry: isTransientError },
);

Why payment writes have a circuit breaker but reads do not:

  • Payment creation is a high-impact operation. If Stripe is down, repeatedly attempting to create payment intents risks duplicate charges when Stripe recovers (idempotency keys help but are not foolproof). The circuit breaker fails fast and signals to the frontend that payments are temporarily unavailable.
  • Read operations (listing payments, searching) are safe to retry without a circuit breaker because they have no side effects. If Stripe's search API is slow, retrying does not risk creating duplicates or charging users.

Timeout: The Stripe SDK is initialised with its own timeout from PAYMENT_CONFIG.stripe.timeout (30 seconds). This acts as the timeout layer, so withTimeout is not applied separately.


Expo Push Notifications ​

File: src/utils/send-push-notification.ts

Configuration:

typescript
const expoCircuitBreaker = new CircuitBreaker({
  name: 'expo-push',
  failureThreshold: 10,
  resetTimeoutMs: 60_000, // 60 seconds
});

const tickets = await expoCircuitBreaker.execute(() =>
  withRetry(
    () =>
      withTimeout(
        _signal => expo.sendPushNotificationsAsync([expoPushMessage]),
        10_000, // 10 second timeout
        'expo-push-notification',
      ),
    { maxRetries: 2, baseDelayMs: 500, shouldRetry: isTransientError },
  ),
);

Why these values:

  • Higher failure threshold (10): Push notifications are fire-and-forget. A missed notification is annoying but not catastrophic. The higher threshold avoids opening the circuit on minor Expo hiccups that resolve quickly.
  • 60s reset: Expo's infrastructure typically recovers within a minute during incidents.
  • 10s timeout: Push delivery should be fast. If Expo is not responding in 10 seconds, something is wrong and we should not hold up the Lambda function.

Error handling: When the circuit is open, the function catches CircuitBreakerOpenError and returns { success: false, error: 'Push notification service temporarily unavailable' } instead of throwing. This prevents push notification failures from breaking the calling operation.


PostgreSQL Database ​

File: src/utils/ppp-db.ts

Configuration:

typescript
const DB_CONFIG = {
  maxPoolSize: 10,
  connectionTimeoutMs: 5_000, // 5 second connection timeout
  idleTimeoutMs: 30_000, // 30 second idle timeout
};

PppDb.pool = postgres(connectionStringUrl, {
  max: DB_CONFIG.maxPoolSize,
  connect_timeout: DB_CONFIG.connectionTimeoutMs / 1000,
  idle_timeout: DB_CONFIG.idleTimeoutMs / 1000,
});

Why this configuration:

  • No circuit breaker: The database is the application's primary data store. If Postgres is down, nothing works. A circuit breaker would just add complexity without a meaningful fallback.
  • No application-level retry: The postgres() constructor is synchronous and creates a pool configuration — it does not connect. Actual connection retries happen at the TCP/TLS level via connect_timeout, and the pool's built-in reconnection logic handles transient failures transparently.
  • 5s connection timeout: Balances between waiting long enough for cold-start DNS resolution and failing fast enough to not burn Lambda execution time.
  • Connection pooling: max: 10 connections per Lambda instance, with a 30-second idle timeout to release unused connections.

Configuration Reference Table ​

ServiceCircuit BreakerRetryTimeoutFallback
Gemini AI5 failures / 60s reset2 retries / 1s base30sFallback model (gemini-1.5-flash)
Stripe (writes)5 failures / 30s reset2 retries / 500ms base30s (SDK config)None
Stripe (reads)None2 retries / 500ms base30s (SDK config)None
Expo Push10 failures / 60s reset2 retries / 500ms base10sNone
PostgreSQLNoneNone (pool-managed)5s (connect)None

Adding Resilience to New Services ​

Follow these steps when integrating a new external service.

Step 1: Assess the service ​

Ask these questions:

  1. Is the call idempotent? If yes, retries are safe. If no, use idempotency keys or skip retries for writes.
  2. Can the app function without this service? If yes, consider a fallback. If no, focus on retries and timeouts.
  3. How critical is latency? Set timeouts based on acceptable response times, not just "how long might it take."
  4. What does failure look like? Identify transient vs permanent error signatures.

Step 2: Add a circuit breaker (if appropriate) ​

Create a module-level circuit breaker instance. This must live outside request handlers so it persists across invocations within the same Lambda container.

typescript
import { CircuitBreaker } from '../../utils/resilience';

// Module-level -- shared across requests in the same Lambda instance
const myServiceCircuitBreaker = new CircuitBreaker({
  name: 'my-service',
  failureThreshold: 5, // Tune based on traffic volume and acceptable false-positive rate
  resetTimeoutMs: 30_000, // Tune based on expected recovery time
});

Step 3: Wrap calls with the resilience stack ​

Apply layers from outermost (circuit breaker) to innermost (timeout):

typescript
import {
  withRetry,
  withTimeout,
  isTransientError,
  CircuitBreakerOpenError,
} from '../../utils/resilience';

const result = await myServiceCircuitBreaker.execute(() =>
  withRetry(
    () =>
      withTimeout(
        signal => myService.call(params, { signal }),
        10_000,
        'my-service-call',
      ),
    {
      maxRetries: 2,
      baseDelayMs: 500,
      shouldRetry: isTransientError,
    },
  ),
);

Step 4: Handle circuit breaker open state ​

Decide what happens when the circuit is open:

typescript
try {
  const result = await myServiceCircuitBreaker.execute(() => ...);
  return result;
} catch (error) {
  if (error instanceof CircuitBreakerOpenError) {
    // Option A: Return a degraded response
    return { data: null, degraded: true };

    // Option B: Return cached data
    return getCachedResult();

    // Option C: Throw a user-friendly error
    throw new TRPCError({
      code: 'SERVICE_UNAVAILABLE',
      message: 'This feature is temporarily unavailable',
    });
  }
  throw error;
}

Step 5: Document the configuration ​

Add a row to the Configuration Reference Table above and explain why you chose those values.


Monitoring and Debugging ​

Log Patterns ​

The resilience utilities emit structured log messages. Search for these patterns in CloudWatch:

PatternMeaning
[Resilience] Attempt N/M failedA retry is about to happen. Includes the error message.
[CircuitBreaker:name] CLOSED -> OPENThe circuit just opened. The service is being rejected.
[CircuitBreaker:name] OPEN -> HALF_OPENThe reset timeout elapsed. A probe request will be tried.
[CircuitBreaker:name] HALF_OPEN -> CLOSEDThe probe succeeded. The service is healthy again.
[CircuitBreaker:name] HALF_OPEN -> OPENThe probe failed. The circuit re-opened.
[CircuitBreaker:name] Reset from X -> CLOSEDSomeone called reset() manually.

Identifying Circuit Breaker Trips in CloudWatch ​

Use CloudWatch Logs Insights:

fields @timestamp, @message
| filter @message like /CircuitBreaker.*OPEN/
| sort @timestamp desc
| limit 50

To correlate with Lambda invocations:

fields @timestamp, @message, @requestId
| filter @message like /CircuitBreaker/ or @message like /\[Resilience\]/
| sort @timestamp desc
| limit 100

Common Failure Scenarios ​

Scenario: Gemini AI outage during peak usage

  1. First 5 OCR requests fail with network errors
  2. Circuit opens: [CircuitBreaker:gemini-ai] CLOSED -> OPEN
  3. Subsequent OCR requests immediately fail with CircuitBreakerOpenError
  4. The fallback model may still work if it is on a different infrastructure
  5. After 60 seconds, a probe request tests Gemini
  6. If the probe succeeds: [CircuitBreaker:gemini-ai] HALF_OPEN -> CLOSED

Scenario: Stripe rate limiting during batch operations

  1. Multiple payment intent creations hit Stripe's rate limit (429)
  2. isTransientError detects 429 and retries with backoff
  3. If retries succeed, no circuit breaker trip
  4. If 5 consecutive calls fail even after retries, circuit opens
  5. Payment endpoints return errors for 30 seconds, then probe

Scenario: Database connection pool exhaustion on cold start

  1. Lambda cold start, no database connections exist
  2. First query triggers a connection attempt via the pool
  3. If the connection fails, the connect_timeout (5s) ensures the attempt does not hang indefinitely
  4. The postgres pool handles reconnection internally on subsequent queries
  5. Once established, the pool is cached for subsequent requests in the same Lambda container

Troubleshooting ​

"Circuit breaker is OPEN" ​

What it means: The circuit breaker for a service has detected enough consecutive failures to conclude the service is down. All requests to that service are being rejected immediately without making network calls.

What to do:

  1. Check the service's status page (Stripe Status, Google Cloud Status, Expo Status).
  2. Search CloudWatch for the transition log: [CircuitBreaker:name] CLOSED -> OPEN to find when the circuit opened and what errors preceded it.
  3. The circuit will automatically transition to HALF_OPEN after the reset timeout elapses and test with a probe request.
  4. If you need to force recovery (e.g., after deploying a fix), redeploy the Lambda function. This creates new instances with fresh circuit breaker state.

"All retry attempts exhausted" ​

What it means: The operation failed on every attempt (initial + retries) and the final error is included in the RetryExhaustedError.

Investigation steps:

  1. Look at the lastError in the log message. This tells you what the underlying failure is.
  2. If the last error is a timeout: the service is responding too slowly. Consider increasing the timeout or reducing the payload.
  3. If the last error is a 429: the service is rate limiting you. Consider reducing request frequency or implementing request queuing.
  4. If the last error is a 5xx: the service has a server-side issue. Check its status page.
  5. If the last error is a connection error (ECONNREFUSED, ECONNRESET): check network connectivity, security groups, and DNS resolution.

"Operation timed out" ​

What it means: The external service did not respond within the configured timeout period.

Tuning timeout values:

  • Start with the p99 latency of the service under normal conditions and add a buffer.
  • For AI inference (Gemini): 30 seconds is appropriate as inference time varies with input size.
  • For simple API calls (Stripe, Expo): 10-15 seconds is usually sufficient.
  • For database connections: 5 seconds covers DNS resolution and TLS handshake.
  • If you see frequent timeouts under normal load, increase the timeout. If timeouts only appear during incidents, the value is correct and the resilience layer is working as designed.

Lambda-Specific Considerations ​

  • Circuit breakers are per-Lambda-instance. Each Lambda container has its own circuit breaker state. If you have 50 concurrent containers, each one independently tracks failures. This means it can take failureThreshold * N total failures across all containers before all circuits are open.
  • Cold starts reset circuit state. A new Lambda container starts with all circuits CLOSED. This is generally fine because a fresh container should probe the service to see if it has recovered.
  • Connection pool is shared within a container. The PppDb class uses a static pool. Once a connection is established in one request, subsequent requests in the same container reuse it.

Internal documentation — org members only