CDC Pipeline Design

Что такое CDC и зачем

Change Data Capture — захват изменений из source database в реальном времени. Вместо periodic full scan — stream отдельных INSERT/UPDATE/DELETE.

Без CDC (Batch):
  Every hour: SELECT * FROM orders WHERE updated_at > last_run
  Проблемы:
    - Пропуск DELETE (нет updated_at)
    - Нагрузка на source DB (full scan)
    - Задержка = 1 час

С CDC (Streaming):
  Real-time: каждый INSERT/UPDATE/DELETE → event в Kafka
  Плюсы:
    - Captures DELETEs
    - Минимальная нагрузка (читает WAL, не таблицы)
    - Задержка = секунды

Log-based vs Query-based CDC

Debezium Architecture

Debezium Event Structure

{
  "before": { "id": 1, "name": "Иван", "city": "Москва" },
  "after":  { "id": 1, "name": "Иван", "city": "Казань" },
  "op": "u",          // c=create, u=update, d=delete, r=read(snapshot)
  "ts_ms": 1700000000,
  "source": {
    "connector": "postgresql",
    "db": "orders_db",
    "schema": "public",
    "table": "customers",
    "lsn": 12345678
  }
}

Exactly-Once CDC

Problem: Debezium restart → may re-send events

Solution layers:
  1. Debezium: tracks LSN/binlog offset in Kafka Connect offsets
  2. Kafka: idempotent producer (dedup by sequence number)
  3. Consumer: idempotent writes
     - MERGE INTO target USING source ON id = id
     - Delta Lake: MERGE with deduplication
     - Iceberg: UPSERT by primary key

End-to-end exactly-once:
  Kafka transactions + idempotent consumer + dedup on write

Schema Evolution в CDC

Source DB: ALTER TABLE customers ADD COLUMN phone VARCHAR(20)

Impact on CDC pipeline:
  1. Debezium detects schema change via WAL
  2. New events include 'phone' field
  3. Schema Registry registers new version (v2)
  4. Consumers must handle both v1 and v2

Strategies:
  a) Forward compatibility: new fields = NULL for old consumers
  b) Schema Registry: enforce BACKWARD/FORWARD compatibility
  c) Bronze layer: store raw (tolerant), validate in Silver