Protobuf Event Evolution: Why buf breaking Isn't Enough

I treated protobufs as stable until a version mismatch ate events. “You can keep the schema compatible, until you realize you need to read an event from 2019.” We discovered this the hard way when a data migration project required replaying five years of order events. The events deserialized without error—Protobuf’s wire format is remarkably stable—but the amount field that used to be in cents was now interpreted as dollars. A $10 order from 2019 looked like a $1,000 order in our new system.

This is the reality of event-driven systems. Events are immutable. Once written, they exist forever in your event store or Kafka topic. You can’t go back and update them. And every time you change your schema, you create a gap between what old events contain and what new code expects.

Buf breaking checks in CI catch syntactic breaking changes: renamed fields, changed types, removed required fields. These are the obvious problems. However, they don’t catch semantic problems—when a field’s meaning changed but its type stayed the same. The schema says int32 amount in both versions, but the interpretation went from cents to dollars. Buf sees no change. Your application sees disaster.

The insidious part is that semantic changes often happen gradually. Someone adds a comment clarifying the field meaning. Months later, new code starts using the “clarified” interpretation. Nobody realizes that old events still use the old interpretation. The bug surfaces years later when someone replays historical events.

Tested on: Protobuf 3, buf CLI 1.28+, Kafka + event store. Production on systems with 3+ years of event history.

Types of Compatibility

There are two fundamentally different kinds of compatibility in event-driven systems, and conflating them is the source of most schema evolution bugs.

Wire Compatibility (Protobuf Guarantees)

Wire compatibility means “can the bytes be deserialized without error?” Protobuf is designed to be very flexible here. New optional fields are ignored by old readers. Missing optional fields get default values in new readers. Type widening (int32 to int64) is handled automatically.

This is what buf breaking checks. It’s essential, but it’s only half the story.

// v1
message OrderCreated {
  string order_id = 1;
  int32 amount = 2;
}

// v2 - wire compatible
message OrderCreated {
  string order_id = 1;
  int64 amount = 2;  // int32 → int64 is OK
  string currency = 3;  // new optional field is OK
}

Semantic Compatibility (You Must Guarantee)

Semantic compatibility means “does the data mean the same thing?” This is your responsibility—Protobuf can’t help you here. The wire format doesn’t encode whether amount is in cents or dollars, whether timestamp is UTC or local time, whether status = 0 means “unknown” or “draft.”

// v1 - amount in cents
message OrderCreated {
  int32 amount = 2;  // 1000 = $10.00
}

// v2 - amount in dollars (BREAKING!)
message OrderCreated {
  int32 amount = 2;  // 1000 = $1000.00 !!!
}

Buf won’t catch this. Old application reads new events and interprets $10 as $1000.

Rules for .proto Files

Never Do

// 1. NEVER change field type incompatibly
string order_id = 1;  →  int64 order_id = 1;  // FORBIDDEN

// 2. NEVER change field number
string order_id = 1;  →  string order_id = 2;  // FORBIDDEN

// 3. NEVER remove required field (proto2)
required string name = 1;  // If exists, must stay

// 4. NEVER rename enum value (if using name matching)
enum Status {
  PENDING = 0;
  ACTIVE = 1;   →   RUNNING = 1;  // FORBIDDEN
}

Safe Changes

// 1. Add optional field
message Order {
  string id = 1;
  string customer_id = 2;  // NEW - OK
}

// 2. Deprecate field (don't remove)
message Order {
  string id = 1;
  string old_field = 2 [deprecated = true];
}

// 3. Extend enum (but watch default)
enum Status {
  UNKNOWN = 0;  // Always have UNKNOWN/UNSPECIFIED = 0
  PENDING = 1;
  ACTIVE = 2;
  CANCELLED = 3;  // NEW - OK
}

// 4. Change optional to repeated (wire compatible)
string tag = 1;  →  repeated string tags = 1;  // OK

Buf CI Setup

buf.yaml

version: v1
breaking:
  use:
    - FILE
  except:
    - FIELD_SAME_JSON_NAME  # Allow JSON name changes
lint:
  use:
    - DEFAULT
  except:
    - PACKAGE_VERSION_SUFFIX

buf.gen.yaml

version: v1
plugins:
  - plugin: buf.build/protocolbuffers/go
    out: gen/go
    opt: paths=source_relative
  - plugin: buf.build/grpc/go
    out: gen/go
    opt: paths=source_relative

GitHub Action

name: Protobuf Check

on: [pull_request]

jobs:
  buf:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: bufbuild/buf-setup-action@v1
        with:
          version: '1.28.0'

      - name: Lint
        run: buf lint

      - name: Breaking change check
        run: buf breaking --against '.git#branch=main'

      - name: Generate
        run: buf generate

      - name: Verify generated files
        run: |
          git diff --exit-code gen/

Upcaster Pattern for Event Evolution

When you inevitably make a breaking semantic change, you need a way to transform old events to the new format. This is where the upcaster pattern comes in. An upcaster is a function that takes an event in an old format and returns it in a new format.

The pattern is borrowed from event sourcing, where you need to read years of historical events with modern code. Instead of maintaining compatibility forever (which limits how much you can evolve your schema), you explicitly handle the transformation.

The key insight is that upcasters form a chain. If you have events from version 1, 2, and 3, and your code expects version 3, you apply upcasters sequentially: v1 → v2 → v3. This means each upcaster only needs to know about two adjacent versions, not the entire history.

When schema changes and you have old events:

interface Event {
  type: string;
  version: number;
  payload: unknown;
  timestamp: Date;
}

interface Upcaster<TOld, TNew> {
  inputVersion: number;
  outputVersion: number;
  upcast(old: TOld): TNew;
}

class OrderCreatedV1ToV2 implements Upcaster<OrderCreatedV1, OrderCreatedV2> {
  inputVersion = 1;
  outputVersion = 2;

  upcast(old: OrderCreatedV1): OrderCreatedV2 {
    return {
      orderId: old.orderId,
      amount: old.amount,
      currency: 'USD',  // Default for old events
      createdAt: old.timestamp,
    };
  }
}

class EventUpcastingService {
  private upcasters: Map<string, Upcaster<any, any>[]> = new Map();

  register(eventType: string, upcaster: Upcaster<any, any>) {
    const chain = this.upcasters.get(eventType) || [];
    chain.push(upcaster);
    chain.sort((a, b) => a.inputVersion - b.inputVersion);
    this.upcasters.set(eventType, chain);
  }

  upcast<T>(event: Event, targetVersion: number): T {
    const chain = this.upcasters.get(event.type) || [];
    let payload = event.payload;
    let currentVersion = event.version;

    for (const upcaster of chain) {
      if (currentVersion >= targetVersion) break;
      if (upcaster.inputVersion === currentVersion) {
        payload = upcaster.upcast(payload);
        currentVersion = upcaster.outputVersion;
      }
    }

    return payload as T;
  }
}

Versioning Strategies

There’s no single right answer for how to version events. Each approach has trade-offs, and the best choice depends on how often you expect breaking changes and how many consumers you have.

Option 1: Version in Package Name

This approach creates separate packages for each major version. It’s explicit and makes breaking changes visible, but requires consumers to handle multiple message types.

The advantage is that you can evolve each version independently. The disadvantage is that consumers need to know about all versions they might encounter, or you need an upcaster layer to normalize everything to the latest version.

// events/order/v1/order_created.proto
package events.order.v1;

message OrderCreated {
  string order_id = 1;
}

// events/order/v2/order_created.proto
package events.order.v2;

message OrderCreated {
  string order_id = 1;
  Money amount = 2;
}

Option 2: Version Field in Event

This approach embeds the version in the event itself. It’s simpler for consumers (one message type) but requires careful handling of optional fields.

The challenge is maintaining backward compatibility as you add versions. Version 3 needs all fields from version 1 and 2, and readers need to know which fields to expect based on the version number.

message OrderCreated {
  int32 schema_version = 1;
  string order_id = 2;

  // V2 fields
  optional Money amount = 3;
}

Option 3: Envelope Pattern

The envelope pattern separates event metadata from the payload. The envelope contains type information, version, timestamp, and routing data. The payload is serialized as Any or raw bytes.

This is the most flexible approach—you can change the payload format without touching the envelope. It’s also the most complex, requiring type registries and dynamic deserialization.

message EventEnvelope {
  string event_type = 1;
  int32 schema_version = 2;
  google.protobuf.Any payload = 3;
  google.protobuf.Timestamp timestamp = 4;
}

Production Checklist

## Protobuf Event Evolution Checklist

### Schema Design
- [ ] Always UNKNOWN = 0 for enums
- [ ] Optional fields for everything new
- [ ] Never delete fields, only deprecate
- [ ] Document semantics in comments

### CI/CD
- [ ] buf lint in every PR
- [ ] buf breaking against main/master
- [ ] Generated files in repo (not runtime)
- [ ] Version tag for schema changes

### Evolution
- [ ] Upcasters for major changes
- [ ] Backward compatibility tests
- [ ] Dual-read period for breaking changes
- [ ] Monitor old versions

### Documentation
- [ ] ADR for each schema change
- [ ] Changelog for .proto files
- [ ] Migration guide for consumers

Conclusion

Schema evolution in event-driven systems is fundamentally different from API evolution. With APIs, you can deprecate and remove endpoints. With events, you can never delete the past. Events written years ago will still be read by code written today.

Protobuf’s wire format compatibility is a necessary foundation, but it’s not sufficient. You need discipline around semantic compatibility—never changing what a field means—and tooling to handle the cases where you must break compatibility.

The upcaster pattern is your escape hatch. When you need to make a breaking change, you don’t try to maintain compatibility in the schema. Instead, you version the event, write an upcaster from old to new, and let your code always work with the latest format. This is more work upfront, but it prevents the subtle bugs that come from misinterpreting old data.

Key principles for sustainable schema evolution:

1. Semantic compatibility is your responsibility - buf can’t catch meaning changes
2. Upcasters convert old events to new format - handle breaking changes explicitly
3. Version explicitly - whether in package names, fields, or envelopes
4. Test with historical data - replay old events through new code as part of CI
5. Document field semantics - comments like “amount in cents” prevent future misunderstandings

FAQ

Can I delete unused field?

No. Mark as deprecated and reserve the number (reserved 5;). Old events may still contain it.

What if I need a breaking change?

Create new event type (OrderCreatedV2), dual-write during migration, then sunset the old one.

How many versions to support?

Minimum: all versions in event store. Practically: 2-3 major versions with upcast chain.

Related Articles

• Transactional Outbox - Event publishing patterns
• API Idempotency - Event deduplication