Droplet ORM
Guide
GitHub
Guide
GitHub
    • Introduction
    • Registry
    • Data Pools
    • Entities
    • Entity Fields
    • MySQL Indexes
    • Code Generation
    • Engine
    • Context
    • Schema Update
    • CRUD Operations
    • Transactions
    • Search
    • Elasticsearch
    • MySQL Queries
    • ClickHouse Queries
    • ClickHouse Schema
    • NATS and JetStream
    • Entity Change Events (CDC)
    • Consumers
    • Tasks and Job Runs
    • Transactional Outbox
    • Context Cache
    • Redis Cache
    • Fake Delete
    • Lifecycle Callbacks
    • Metrics
    • Redis Operations
    • Distributed Lock
    • Queries Log
    • Testing
    • Migrating from fluxaorm

Entity Change Events (CDC)

An entity tagged orm:"cdc" puts a change event on a NATS JetStream change feed every time a row is inserted, updated or deleted through Droplet ORM. Consumers receive those events as typed before/after entities and use them to refresh search indexes, send notifications, keep projections in sync - anything that must react to a change without slowing down the write path.

The path of one change:

ctx.Save(entity)
   │  after commit
   ▼
stream CDC          subject cdc.<database>.<table>      Debezium MySQL envelope
   │  CDC reader (orm.NewCDCReader, durable orm-cdc-reader)
   ▼
stream ORM_ENTITY   subject orm.entity.<table>          same bytes
   │  one durable per ConsumerDef
   ▼
On<Entity>(ctx, ev *entities.<Entity>DirtyEvent)

Two things follow from this shape:

  • The feed speaks the Debezium MySQL envelope, value encodings included. Droplet ORM writes it itself after commit; a Debezium connector can take over the same subjects later (External producer) and nothing that reads the feed can tell the two apart.
  • The entity knows nothing about who reads it; each consumer declares which entities it wants. One write is one message however many consumers read it - fan-out happens on the subscribe side.

Enabling Events

Put cdc in the orm tag of the ID field:

type UserEntity struct {
    ID    uint64 `orm:"cdc"`
    Email string `orm:"required;length=255"`
    Name  string `orm:"required;length=100"`
}

Four more things are required:

  1. A NATS pool. The entity stream lives on the pool registered as orm.DefaultPoolCode unless RegisterEntityStream picks another; the change feed uses the same pool unless RegisterCDCStream picks another. Validate() does not check the pool exists - a missing one surfaces at the first Save as nats pool '<code>' for the cdc stream not configured.
  2. At least one ConsumerDef that declares the entity. An event nobody reads is pure cost, so Validate() fails with: entity 'UserEntity' is tagged orm:"cdc" but no consumer declares it, so its events would be published and never read; add it to a ConsumerDef or drop the tag
  3. Generated code (see Code generation). The generator emits the per-entity row codec and registers the entity on the feed; until it has run once, nothing is published (this is what lets the very first Generate succeed).
  4. A running CDC reader. Without it changes pile up on the CDC stream and no consumer sees them.
registry := orm.NewRegistry()
registry.RegisterMySQL("user:password@tcp(localhost:3306)/app", orm.DefaultPoolCode, &orm.MySQLOptions{})
registry.RegisterNats([]string{"nats://localhost:4222"}, orm.DefaultPoolCode, nil)
registry.RegisterEntity(UserEntity{})
registry.RegisterConsumer(orm.ConsumerDef{
    Name:     "user-indexer",
    Entities: []any{UserEntity{}},
})
engine, err := registry.Validate()

Removed tag

The old orm:"dirty=stream1,stream2" form no longer exists and is rejected at Validate(): entity 'UserEntity' uses orm:"dirty=...", which no longer exists; tag it orm:"cdc" and declare the entity on a orm.ConsumerDef instead

The Change Feed

Every change of a cdc entity is published to the stream CDC (orm.CDCStreamName, subjects cdc.>) on its own subject:

cdc.<database>.<table name>        e.g. cdc.app.UserEntity

<database> is the database name of the entity's MySQL pool and <table name> is the table name, not the Go type name. orm.CDCSubject(database, table) builds it. This is the naming a Debezium connector with topic prefix cdc uses.

The stream is tuned with RegisterCDCStream (optional - the defaults apply either way):

registry.RegisterCDCStream(orm.CDCStreamOptions{
    MaxBytes:      8 << 30,
    ReaderAckWait: time.Minute,
})
FieldTypeDefault
Producerorm.CDCProducerorm.CDCProducerORM - see External producer
NatsPoolstringthe entity stream's pool (EntityStreamOptions.NatsPool), else orm.DefaultPoolCode
Storagejetstream.StorageTypefile storage
Replicasint1
MaxAgetime.Duration7 * 24 * time.Hour
MaxBytesint64unbounded
DuplicateWindowtime.Duration2 * time.Minute
ReaderAckWaittime.Duration30 * time.Second - must exceed the slowest forward of one fetched batch
ReaderMaxAckPendingint256

GetNatsAlters creates the CDC stream when it is missing and never updates or deletes it: once a Debezium deployment writes to the stream it owns the limits, and the ORM must not fight it. Changing Storage, Replicas, MaxAge, MaxBytes or DuplicateWindow after the stream exists therefore has no effect; change the stream on the broker. The reader's durable (ReaderAckWait, ReaderMaxAckPending, the list of tables) is reconciled on every run.

The envelope

{
  "before": {"ID": 323046535625160701, "Email": "old@example.com", "Name": "Ann"},
  "after":  {"ID": 323046535625160701, "Email": "new@example.com", "Name": "Ann"},
  "op": "u",
  "ts_ms": 1757600000000,
  "source": {"connector": "orm", "db": "app", "table": "UserEntity", "ts_ms": 1757600000000}
}
opConstantWritten forbeforeafter
corm.CDCOpCreateinsertnullnew row
uorm.CDCOpUpdateupdate, and a soft delete (an UPDATE that sets FakeDelete, exactly as the binlog sees it)old rownew row
dorm.CDCOpDeletehard deleteold rownull
rorm.CDCOpReadDebezium snapshot reads, and replaysnullcurrent row

Row images are keyed by column name and hold every column of the table. Values use Debezium's encodings:

ColumnJSON value
integers, references, BIGINT UNSIGNEDnumber
tinyint(1) (Go bool)boolean (0/1 is accepted when reading)
FakeDeletenumber (0 or the row's ID)
FLOAT / DOUBLEnumber
DECIMAL (orm:"decimal=...")string at the column's scale, e.g. "12.50"
DATETIME (orm:"time")epoch milliseconds
DATEepoch days
strings, enums, sets, References[T], JSON structsstring
[]uint8 (blob)base64 string
NULLnull

You rarely read the envelope yourself: consumers get it decoded into entities. It matters when another system reads the CDC stream directly, or when Debezium writes it.

The Entity Stream

Listeners do not read the CDC stream. They read one JetStream stream, ORM_ENTITY (orm.EntityStreamName), capturing subjects orm.entity.> and orm.replay.>. The CDC reader forwards each change there, on the entity's own subject:

orm.entity.<table name>        e.g. orm.entity.UserEntity

The token is the table name, not the Go type name, so renaming the struct without changing orm:"table=..." does not move the subject. orm.EntitySubject(table) builds it.

The stream is created and kept in sync by GetNatsAlters. It is tuned with RegisterEntityStream (optional - the stream exists with defaults either way):

registry.RegisterEntityStream(orm.EntityStreamOptions{
    NatsPool: "events",
    MaxBytes: 4 << 30,
})
FieldTypeDefault
NatsPoolstringorm.DefaultPoolCode ("default")
Storagejetstream.StorageTypefile storage
Replicasint1
MaxAgetime.Duration7 * 24 * time.Hour
MaxBytesint64unbounded
DuplicateWindowtime.Duration10 * time.Minute

Tips

Set MaxBytes explicitly in production. Replay shares this stream, so a large reindex on an unbounded stream either evicts live events or fills the disk.

The CDC Reader

func NewCDCReader(engine Engine) StreamConsumer

The reader is the one consumer of the change feed: the durable orm-cdc-reader (orm.CDCReaderName) on CDC, filtering the subject of every cdc table. Like every StreamConsumer it has no blocking Run() - the application owns the loop:

reader := orm.NewCDCReader(engine)
go func() {
    for appCtx.Err() == nil {
        if err := reader.Consume(appCtx, 64, time.Second); err != nil && appCtx.Err() == nil {
            log.Printf("cdc reader: %v", err)
            time.Sleep(time.Second) // back off; the batch is redelivered
        }
    }
}()

Per fetched batch, Consume:

  1. Forwards every c, u and d event to orm.entity.<table> on ORM_ENTITY, bytes unchanged and in fetch order across tables, in one PublishBatch. A listener reading several entities therefore sees their changes in the order they were written.
  2. Acks the whole batch.
SituationResult
r event (Debezium snapshot)Acked, not forwarded. A snapshot read is not a change; re-driving listeners is what replay is for.
Forward failsNothing is acked and Consume returns forward <n> cdc events: <cause>, so the loop backs off instead of fetching past the changes it could not deliver. The batch is redelivered after ReaderAckWait.
Bytes that do not parse as an envelopeLogged (dropped undecodable cdc event) and acked, so they cannot block the feed.
NATS pool missingnats pool '<code>' for the cdc reader not configured

Run one reader loop: forwarding in fetch order is an ordering guarantee only with a single fetcher. ConsumerPending(ctx, orm.CDCReaderName) returns the reader's backlog, see Monitoring.

Listener Events

type DirtyOp uint8

const (
    DirtyInsert DirtyOp = 1
    DirtyUpdate DirtyOp = 2
    DirtyDelete DirtyOp = 3
)
// (DirtyOp).String() returns "insert", "update", "delete" or "unknown"

type DirtyEvent[T any] struct {
    Op     DirtyOp
    ID     uint64
    Before *T // nil for Insert
    After  *T // nil for Delete
    TsMs   int64
}

For every cdc (or outbox) entity the generator emits a type alias in the entities package:

type UserEntityDirtyEvent = orm.DirtyEvent[UserEntity]

Before and After are read-only generated entities decoded from the row images. Read them through the ordinary typed getters:

func(ctx orm.Context, ev *entities.UserEntityDirtyEvent) error {
    switch ev.Op {
    case orm.DirtyDelete:
        return search.Remove(ev.ID)
    case orm.DirtyUpdate:
        if ev.Before.GetEmail() != ev.After.GetEmail() {
            return mailer.ConfirmNewAddress(ev.ID, ev.After.GetEmail())
        }
    }
    return search.Index(ev.ID, ev.After.GetEmail(), ev.After.GetName())
}
  • They are snapshots of the row at the time of the change, not live handles: ctx.Save, ctx.Delete, ctx.ForceDelete and ctx.Reload reject them with orm.ErrEntityReadOnly. To write, load the live entity with <Entity>Provider.GetByID(ctx, ev.ID).
  • Reference getters (ev.After.GetCategory(ctx)) work and load through the handler's context.
  • They are not placed in the context cache.
  • A column missing from an image decodes as its zero value - that is what a column added after the event was written looks like.

How the envelope maps to an event:

EnvelopeOpBeforeAfter
cDirtyInsertnilnew row
uDirtyUpdateold rownew row
u that sets FakeDelete (soft delete)DirtyDeleteold rownil
dDirtyDeleteold rownil
r (replay)DirtyUpdatecurrent rowthe same row

ID is taken from the row. TsMs is source.ts_ms (falling back to the envelope's ts_ms): the binlog commit time when Debezium produces the feed, the time the ORM serialised the event right around the commit otherwise.

An envelope with an unknown op or without a row is a decode error; the handler is not called and the message is redelivered (see What happens to a message).

When Events Are Published

With the default producer, publishing is part of ctx.Save() (see CRUD) and happens after the database changes are durable:

  1. Entity statements (and outbox rows, if any) are executed on the database pipeline.
  2. Outside a transaction the post-commit step runs immediately; inside ctx.Transaction(...) it is queued and runs after the commit (see Transactions).
  3. Post-commit order: evict handles that remain deleted from the context cache, cache invalidation, Redis pipelines, change feed publish, outbox mark, after-handlers.

Each write captures its own row images. If a new entity is saved with Name="first", changed to "second" and saved again in one transaction, commit publishes a c with after.Name="first", then a u with before.Name="first" and after.Name="second". Later edits to the live entity cannot change these payloads. An unchanged Save produces no event, and rollback publishes none.

All events of one Save call are published to the CDC stream with a single PublishBatch, and the post-commit phase waits for the JetStream acks. Outside a transaction, Save returning nil means the event is durably stored on the change feed. Inside a transaction, that guarantee applies only when the outermost Transaction returns nil; an individual tx.Save returns before publication.

If the publish fails after the rows were committed, Save (or Transaction) returns a *orm.PostCommitError whose message starts with post-commit failure (database changes are committed): and wraps publish <n> entity events to pool <pool>: <cause>:

u := entities.UserEntityProvider.New(ctx)
u.SetEmail("ann@example.com").SetName("Ann")
if err := ctx.Save(u); err != nil {
    var postCommit *orm.PostCommitError
    if errors.As(err, &postCommit) {
        // The row IS in the database; only the event did not reach NATS.
        // Saving this unchanged entity again does not retry publication.
        // See the outbox page for durable event delivery.
    }
    return err
}

Guaranteed delivery

cdc alone is "publish after commit": a NATS outage in that window loses the event. Add outbox to the tag (orm:"cdc;outbox") to store the event in the database inside the write's own transaction and have a relay republish it - see Transactional outbox. With Debezium as the producer the binlog is the guarantee and no outbox is needed.

Deduplication

Every message on the feed carries Nats-Msg-Id = <subject>:<FNV-1a 64 hash of the payload>. JetStream drops a second message with the same id inside the stream's DuplicateWindow. Two places rely on it:

  • CDC stream (2 minutes by default): the outbox relay republishes the stored bytes under the same id, so a republish after a failure is a no-op when the original did arrive.
  • ORM_ENTITY stream (10 minutes by default): the reader's forward carries the id for orm.entity.<table> and the same payload, so a batch that is redelivered to the reader is not delivered twice to listeners.

The id contains no consumer name, which is what makes one write one message however many consumers read it. Replays use a different, unique id on purpose.

External Producer (Debezium)

registry.RegisterCDCStream(orm.CDCStreamOptions{Producer: orm.CDCProducerExternal})

With orm.CDCProducerExternal the ORM publishes nothing after a write and writes no outbox rows; a Debezium MySQL connector is expected to write the same subjects from the binlog. The reader, the entity stream, consumers, handlers and replay are unchanged - that is the point of the shared format.

The connector must produce what the ORM produces:

  • topic prefix cdc, so subjects are cdc.<database>.<table> on the CDC stream
  • JSON values without the schema wrapper (format.value=json, format.value.schemas.enable=false)
  • decimal.handling.mode=string
  • bigint.unsigned.handling.mode=long
  • time.precision.mode=adaptive_time_microseconds
  • binary.handling.mode=base64
  • the TinyIntOneToBooleanConverter for tinyint(1) columns
  • tombstones.on.delete=false

These are the settings the ORM's parity test runs Debezium Server with (docker/debezium/application.properties in the ORM repository).

Differences a listener can observe after the switch: changes made outside the ORM (manual SQL, other services) now produce events too, and TsMs is the binlog commit time. FLOAT (32-bit) columns may differ in the last digits between the two producers; DOUBLE and DECIMAL do not.

Generated Code

For each cdc/outbox entity the generated entity file contains:

  • the <Entity>DirtyEvent alias;
  • the row image codec (an unexported column list and decoder) and the methods the runtime calls on the entity - PrivateCDCRow, PrivateCDCChange, PrivateColumnValue - which, like every Private* method, are not for application code;
  • <Entity>Provider.NewSnapshot(id, fill), a read-only entity built in memory for tests;
  • an init() that registers the entity on the feed under its table name:
func init() {
    orm.RegisterCDCEntity[UserEntity]("UserEntity")
}

You never call RegisterCDCEntity yourself. At Validate() Droplet ORM matches each cdc/outbox schema to its registration by the generated entity name; a missing one is tolerated so that the first code generation can run. The generated code never spells out a subject - the format belongs to Droplet ORM.

Monitoring

SignalMeaning
orm.ConsumerPending(ctx, orm.CDCReaderName)Changes on the feed the reader has not forwarded yet. Growing means the reader loop is not running or cannot publish.
orm.ConsumerPending(ctx, entities.ConsumerUserIndexer.Name())Backlog of one listener on ORM_ENTITY.
orm_cdc_messages_total{consumer="orm-cdc-reader",entity,op}Events the reader fetched; op is insert, update, delete, read or unknown, taken from the envelope's op (a soft delete counts as update).
orm_cdc_messages_total{consumer="<name>",entity,op}Events a declared consumer fetched.
orm_stream_consume_lag_seconds{consumer}Broker timestamp to fetch, for the reader and for every consumer. End-to-end latency is the sum of both hops.
orm_nats_operations_seconds{operation="publish"}, orm_nats_publish_batch_sizeThe post-commit publish and the reader's forwards.

Register a query logger with orm.QueryLoggerOptions{Nats: true} to see every publish (PUBLISH_BATCH), the reader's errors (CDC_READ) and handler errors (CDC_HANDLE). See NATS and JetStream and Metrics.

Complete Example

package main

import (
    "context"
    "errors"
    "fmt"
    "log"
    "os/signal"
    "syscall"
    "time"

    "github.com/DropletLive/orm"

    "myapp/entities"
)

type UserEntity struct {
    ID    uint64 `orm:"cdc"`
    Email string `orm:"required;length=255"`
    Name  string `orm:"required;length=100"`
}

func main() {
    registry := orm.NewRegistry()
    registry.RegisterMySQL("user:password@tcp(localhost:3306)/app", orm.DefaultPoolCode, &orm.MySQLOptions{})
    registry.RegisterRedis("localhost:6379", 0, orm.DefaultPoolCode, nil)
    registry.RegisterNats([]string{"nats://localhost:4222"}, orm.DefaultPoolCode, nil)
    registry.RegisterEntityStream(orm.EntityStreamOptions{MaxBytes: 4 << 30})
    registry.RegisterCDCStream(orm.CDCStreamOptions{MaxBytes: 4 << 30})
    registry.RegisterEntity(UserEntity{})
    registry.RegisterConsumer(orm.ConsumerDef{Name: "user-indexer", Entities: []any{UserEntity{}}})

    engine, err := registry.Validate()
    if err != nil {
        panic(err)
    }
    defer engine.Nats(orm.DefaultPoolCode).Close()

    appCtx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
    defer stop()
    ctx := engine.NewContext(appCtx)

    // Tables, then JetStream streams (CDC, ORM_ENTITY) and durables.
    alters, err := orm.GetAlters(ctx)
    if err != nil {
        panic(err)
    }
    for _, alter := range alters {
        if err := alter.Exec(ctx); err != nil {
            panic(err)
        }
    }
    natsAlters, err := orm.GetNatsAlters(ctx)
    if err != nil {
        panic(err)
    }
    for _, alter := range natsAlters {
        if err := alter.Exec(ctx); err != nil {
            panic(err)
        }
    }

    // CDC reader: forwards cdc.app.UserEntity to orm.entity.UserEntity.
    reader := orm.NewCDCReader(engine)
    go loop(appCtx, "cdc-reader", reader, 64)

    // Consumer: every UserEntity change lands here (see the Consumers page).
    indexer := orm.NewConsumer(engine, entities.ConsumerUserIndexer).
        OnUserEntity(func(ctx orm.Context, ev *entities.UserEntityDirtyEvent) error {
            if ev.Op == orm.DirtyDelete {
                fmt.Printf("user %d deleted\n", ev.ID)
                return nil
            }
            fmt.Printf("user %d %s: %s\n", ev.ID, ev.Op, ev.After.GetEmail())
            return nil
        }).
        Build()
    go loop(appCtx, "user-indexer", indexer, 32)

    // One Save = one row + one event on the change feed.
    u := entities.UserEntityProvider.New(ctx)
    u.SetEmail("ann@example.com").SetName("Ann")
    if err := ctx.Save(u); err != nil {
        var postCommit *orm.PostCommitError
        if errors.As(err, &postCommit) {
            log.Printf("row committed but event not published: %v", err)
        } else {
            panic(err)
        }
    }

    <-appCtx.Done()
}

func loop(ctx context.Context, name string, consumer orm.StreamConsumer, batch int) {
    for ctx.Err() == nil {
        if err := consumer.Consume(ctx, batch, time.Second); err != nil && ctx.Err() == nil {
            log.Printf("%s: %v", name, err)
            time.Sleep(time.Second)
        }
    }
}
Edit this page
Last Updated: 11/16/26, 1:49 PM
Prev
NATS and JetStream
Next
Consumers