API Idempotency: Designing Endpoints Resistant to Retries

The first double-charge I saw was in a happy-path deployment, and that is when I stopped trusting retries. “Customer paid twice!” I heard this at 7:00 AM after deploying a new version of the payment system. Client clicked “Pay”, their Wi-Fi dropped, browser automatically retried the request, and we processed the payment twice.

Problem? Our POST endpoint wasn’t idempotent. Solution? Idempotency-Key pattern.

Tested on: Node.js/TypeScript, Redis 7.x, PostgreSQL 15+. Production-proven on payment system handling 10k+ transactions daily.

What is Idempotency

Idempotent operation - if you execute it multiple times, the result is the same as if you executed it once.

GET /users/123        ✓ Idempotent (read)
PUT /users/123        ✓ Idempotent (replace)
DELETE /users/123     ✓ Idempotent (already deleted = OK)
POST /orders          ✗ NOT idempotent (creates new)
PATCH /users/123      ⚠ Depends on implementation

Why POST is Not Automatically Idempotent

// Naive implementation
app.post('/orders', async (req, res) => {
  const order = await db.orders.create(req.body);
  await paymentService.charge(order.total);
  res.json(order);
});

// What happens on retry?
// 1. Request → DB insert → Payment → 200 OK
// 2. Retry   → DB insert → Payment → 200 OK  // DUPLICATE!

Idempotency-Key Pattern

Principle

1. Client generates unique Idempotency-Key for each unique request
2. Server stores key + response on first processing
3. On retry, server returns stored response

Client                          Server
  │                               │
  ├── POST /orders ──────────────►│
  │   Idempotency-Key: abc123     │
  │                               ├── Process request
  │                               ├── Store (key, response)
  │◄── 200 OK ────────────────────┤
  │                               │
  │   (Network failure, retry)    │
  │                               │
  ├── POST /orders ──────────────►│
  │   Idempotency-Key: abc123     │
  │                               ├── Find stored response
  │◄── 200 OK (cached) ───────────┤

TypeScript Implementation

// middleware/idempotency.ts
import { Redis } from 'ioredis';

interface IdempotencyRecord {
  status: 'processing' | 'completed' | 'failed';
  response?: {
    statusCode: number;
    body: unknown;
    headers: Record<string, string>;
  };
  createdAt: number;
  expiresAt: number;
}

const IDEMPOTENCY_TTL = 24 * 60 * 60; // 24 hours

export function idempotencyMiddleware(redis: Redis) {
  return async (req: Request, res: Response, next: NextFunction) => {
    // Only for POST/PATCH requests
    if (!['POST', 'PATCH'].includes(req.method)) {
      return next();
    }

    const idempotencyKey = req.headers['idempotency-key'] as string;

    if (!idempotencyKey) {
      return res.status(400).json({
        error: 'Missing Idempotency-Key header'
      });
    }

    const cacheKey = `idempotency:${req.path}:${idempotencyKey}`;

    // Check if already exists
    const existing = await redis.get(cacheKey);

    if (existing) {
      const record: IdempotencyRecord = JSON.parse(existing);

      if (record.status === 'processing') {
        // Request is being processed
        return res.status(409).json({
          error: 'Request is still being processed',
          retryAfter: 1
        });
      }

      if (record.status === 'completed' && record.response) {
        // Return stored response
        res.set('Idempotency-Replayed', 'true');
        Object.entries(record.response.headers).forEach(([k, v]) => {
          res.set(k, v);
        });
        return res.status(record.response.statusCode).json(record.response.body);
      }

      if (record.status === 'failed') {
        // Previous attempt failed, allow retry
        // (continue processing)
      }
    }

    // Mark as "processing"
    const processingRecord: IdempotencyRecord = {
      status: 'processing',
      createdAt: Date.now(),
      expiresAt: Date.now() + IDEMPOTENCY_TTL * 1000
    };

    // Atomic set with NX (only if doesn't exist)
    const set = await redis.set(
      cacheKey,
      JSON.stringify(processingRecord),
      'NX',
      'EX',
      IDEMPOTENCY_TTL
    );

    if (!set) {
      // Someone else is processing
      return res.status(409).json({
        error: 'Concurrent request detected',
        retryAfter: 1
      });
    }

    // Intercept response
    const originalJson = res.json.bind(res);
    res.json = (body: unknown) => {
      const completedRecord: IdempotencyRecord = {
        status: res.statusCode >= 400 ? 'failed' : 'completed',
        response: {
          statusCode: res.statusCode,
          body,
          headers: Object.fromEntries(
            Object.entries(res.getHeaders())
              .filter(([k]) => ['content-type', 'x-request-id'].includes(k.toLowerCase()))
              .map(([k, v]) => [k, String(v)])
          )
        },
        createdAt: processingRecord.createdAt,
        expiresAt: processingRecord.expiresAt
      };

      // Store result (async, don't block response)
      redis.set(
        cacheKey,
        JSON.stringify(completedRecord),
        'EX',
        IDEMPOTENCY_TTL
      ).catch(console.error);

      return originalJson(body);
    };

    next();
  };
}

Usage

const app = express();
const redis = new Redis(process.env.REDIS_URL);

app.use(idempotencyMiddleware(redis));

app.post('/orders', async (req, res) => {
  // This handler executes only once per Idempotency-Key
  const order = await createOrder(req.body);
  await chargePayment(order);
  res.json(order);
});

Processing State Diagram

┌─────────────────────────────────────────────────────────────┐
│                                                             │
│   ┌──────────┐    key exists?    ┌──────────────┐          │
│   │  START   │───────────────────│ Check Status │          │
│   └──────────┘        no         └──────────────┘          │
│        │                               │                    │
│        │                    ┌──────────┴──────────┐        │
│        │                    │                     │        │
│        ▼              processing           completed       │
│   ┌──────────┐              │                     │        │
│   │  Lock    │              ▼                     ▼        │
│   │ (NX SET) │         ┌──────────┐         ┌──────────┐  │
│   └──────────┘         │  409     │         │  Return  │  │
│        │               │ Conflict │         │  Cached  │  │
│        │               └──────────┘         └──────────┘  │
│        ▼                                                   │
│   ┌──────────┐                                             │
│   │ Process  │                                             │
│   │ Request  │                                             │
│   └──────────┘                                             │
│        │                                                   │
│   ┌────┴────┐                                              │
│   │         │                                              │
│ success   error                                            │
│   │         │                                              │
│   ▼         ▼                                              │
│ ┌────┐   ┌────────┐                                        │
│ │Save│   │ Save   │                                        │
│ │ OK │   │ Failed │                                        │
│ └────┘   └────────┘                                        │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Where to Store Idempotency Keys

Redis (Recommended)

// Pros: fast, automatic expiry, atomic operations
const redis = new Redis({
  host: process.env.REDIS_HOST,
  keyPrefix: 'idempotency:',
  retryStrategy: (times) => Math.min(times * 50, 2000)
});

PostgreSQL (Alternative)

CREATE TABLE idempotency_keys (
  key VARCHAR(255) PRIMARY KEY,
  path VARCHAR(500) NOT NULL,
  status VARCHAR(20) NOT NULL,
  response JSONB,
  created_at TIMESTAMP DEFAULT NOW(),
  expires_at TIMESTAMP NOT NULL
);

CREATE INDEX idx_idempotency_expires ON idempotency_keys(expires_at);

-- Cleanup job
DELETE FROM idempotency_keys WHERE expires_at < NOW();

Comparison

Best Practices

1. Generate Idempotency-Key on Client

// Frontend
async function createOrder(orderData: OrderInput) {
  const idempotencyKey = crypto.randomUUID();

  return fetch('/api/orders', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Idempotency-Key': idempotencyKey
    },
    body: JSON.stringify(orderData)
  });
}

2. Key Should Be Tied to Content

// Better: hash from request content
const idempotencyKey = crypto
  .createHash('sha256')
  .update(JSON.stringify({ userId, items, timestamp: Date.now() }))
  .digest('hex');

3. Retry with Exponential Backoff

async function fetchWithRetry(url: string, options: RequestInit, maxRetries = 3) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      const response = await fetch(url, options);

      // 409 = retry later
      if (response.status === 409) {
        const retryAfter = parseInt(response.headers.get('Retry-After') || '1');
        await delay(retryAfter * 1000);
        continue;
      }

      return response;
    } catch (error) {
      if (attempt === maxRetries - 1) throw error;
      await delay(Math.pow(2, attempt) * 1000);
    }
  }
}

4. Log Idempotency-Key for Debugging

app.use((req, res, next) => {
  const idempotencyKey = req.headers['idempotency-key'];
  if (idempotencyKey) {
    req.log = req.log.child({ idempotencyKey });
  }
  next();
});

Production Checklist

## Idempotency Implementation Checklist

### Middleware
- [ ] Validate Idempotency-Key presence for POST/PATCH
- [ ] Atomic lock (SET NX in Redis or INSERT...ON CONFLICT)
- [ ] Handle "processing" state (409 Conflict)
- [ ] Return cached response with Idempotency-Replayed header
- [ ] TTL on keys (24-48h)

### Storage
- [ ] Redis/PostgreSQL with retries
- [ ] Cleanup expired keys (cron or TTL)
- [ ] Monitor storage size

### Client
- [ ] Generate unique key for each logical request
- [ ] Retry with exponential backoff
- [ ] Handle 409 Conflict gracefully

### Monitoring
- [ ] Metric: idempotency cache hit rate
- [ ] Metric: processing conflicts (409s)
- [ ] Alert: high conflict rate (possible bug)
- [ ] Log: idempotencyKey for each request

Special Cases

Partial Failure

app.post('/orders', async (req, res) => {
  const order = await db.orders.create(req.body);  // ✓ Success

  try {
    await paymentService.charge(order.total);      // ✗ Failure
  } catch (error) {
    // What now? Order exists, payment doesn't.
    // Retry will create another order!

    // Solution: Transaction or Saga pattern
    await db.orders.delete(order.id);
    throw error;
  }

  res.json(order);
});

Async Processing

app.post('/reports', async (req, res) => {
  // Report generates asynchronously (5+ minutes)
  const job = await queue.add('generate-report', req.body);

  // Return job ID, not result
  res.status(202).json({
    jobId: job.id,
    status: 'processing',
    checkUrl: `/reports/${job.id}`
  });

  // Client polls status
});

Conclusion

Idempotency is a fundamental building block for reliable APIs. Without it, retries are dangerous and can cause duplicates, inconsistencies, and financial losses.

Key principles:

1. Every POST/PATCH needs Idempotency-Key
2. Atomic locking prevents race conditions
3. Cached responses ensure consistent answers
4. Reasonable TTL (24-48h) balances storage and usability
5. Client retry logic with exponential backoff

FAQ

How long should Idempotency-Key be?

UUID (36 characters) is ideal. Longer hashes (SHA-256) are also OK. Shorter increases collision risk.

What if Redis is unavailable?

Fail open (allow request without idempotency) or fail closed (return 503). Depends on use case - payments should fail closed.

How long to keep keys?

24-48 hours is standard. For payments, possibly longer. Too long = too much storage.

What if client sends different body with same key?

Ignore new body, return original response. Key is a “promise” - same key = same operation.

Related Articles

• Transactional Outbox - Reliable messaging pattern
• The Soft Delete Trap - Idempotent delete operations