Droplet ORM
Guide
GitHub
Guide
GitHub
    • Introduction
    • Registry
    • Data Pools
    • Entities
    • Entity Fields
    • MySQL Indexes
    • Code Generation
    • Engine
    • Context
    • Schema Update
    • CRUD Operations
    • Transactions
    • Search
    • Elasticsearch
    • MySQL Queries
    • ClickHouse Queries
    • ClickHouse Schema
    • NATS and JetStream
    • Entity Change Events (CDC)
    • Consumers
    • Tasks and Job Runs
    • Transactional Outbox
    • Context Cache
    • Redis Cache
    • Fake Delete
    • Lifecycle Callbacks
    • Metrics
    • Redis Operations
    • Distributed Lock
    • Queries Log
    • Testing
    • Migrating from fluxaorm

NATS and JetStream

Droplet ORM uses NATS JetStream as its messaging transport. Three features are built on top of it:

  • Entity change events - every write to an entity tagged orm:"cdc" goes on the CDC change feed and is forwarded to consumers.
  • Consumers - typed, generated handlers that read entity events and tasks.
  • Tasks - background jobs dispatched onto queues, with a run history table.

A NATS pool is a connection to one or more NATS servers. This page covers the pool itself, the low-level publish and fetch API, hand-registered streams and consumers, and the topology reconciliation (GetNatsAlters) that creates and updates streams and durable consumers on the broker.

Pool Registration

Register a pool with RegisterNats:

import "github.com/DropletLive/orm"

registry := orm.NewRegistry()

// Pool with default options (nil is allowed and becomes an empty NatsPoolOptions):
registry.RegisterNats([]string{"nats://localhost:4222"}, orm.DefaultPoolCode, nil)

// Pool with several servers and options:
registry.RegisterNats([]string{"nats://nats-1:4222", "nats://nats-2:4222"}, "events", &orm.NatsPoolOptions{
    ClientID:             "my-service",
    RetryOnFailedConnect: true,
    Auth: &orm.NatsAuthConfig{
        User:     "app",
        Password: "secret",
    },
})
RegisterNats(urls []string, poolCode string, options *NatsPoolOptions)

The pool is available from the engine as engine.Nats(code) (returns nil for an unknown code) and all pools as engine.Registry().NatsPools().

Lazy connection

registry.Validate() only builds the pool object; nothing connects to NATS until the first operation that needs a connection (Publish*, Consumer, GetJetStream, GetConn, Ping). Ping() forces a connection attempt and is the way to fail fast at startup.

The features built on NATS pick their pool with the NatsPool field of EntityStreamOptions, CDCStreamOptions, TaskStreamOptions and ConsumerDef. They default to orm.DefaultPoolCode ("default") - CDCStreamOptions.NatsPool to the entity stream's pool - so registering the pool under the default code needs no further wiring.

Connection lifecycle

A pool owns one connection and one JetStream context, shared by every publish and every consumer made from it.

  • A reconnecting connection is kept. When the server goes away the NATS client reconnects by itself (MaxReconnects, ReconnectWait). The pool hands out the same connection the whole time; operations issued while it is down fail or wait according to the client's own rules, and work again when it is back. Nothing made from the pool has to be recreated.
  • A closed connection is replaced. If the connection is closed - the reconnect attempts ran out with a finite MaxReconnects, or Close() was called - the next operation dials a new one.
  • Consumers follow the pool. A NatsConsumer (and every StreamConsumer built on one) asks the pool for the current JetStream context on each Fetch and re-resolves its durable when the context changed. A consumer obtained once at startup keeps working across reconnects and across a replaced connection; a fetch loop only has to log the error, back off and call Fetch/Consume again.
  • GetConn() and GetJetStream() return the current connection. Do not cache them across a Close(), and do not close the returned *nats.Conn yourself - call pool.Close().

NatsPoolOptions

FieldTypeBehaviour
ClientIDstringPassed as the connection name (nats.Name).
MaxReconnectsintDefault -1 (reconnect forever). Applied only when not 0.
ReconnectWaittime.DurationDefault 1s. Applied when > 0.
ReconnectBufSizeintApplied when > 0.
PingIntervaltime.DurationApplied when > 0.
ConnectTimeouttime.DurationApplied when > 0 (nats.Timeout).
RetryOnFailedConnectboolWhen true the client keeps retrying if the initial connect fails.
Auth*NatsAuthConfigAuthentication, see below.
IgnoredSubjects[]stringExisting ORM_*/orm_* streams carrying any of these subjects are never deleted by GetNatsAlters. The list is only taken into account when at least one stream is registered with RegisterNatsStream.
IgnoredConsumers[]stringParsed and stored, but not read anywhere - GetNatsAlters never deletes consumers, so there is nothing to exempt. Setting it has no effect today.

NatsAuthConfig

Exactly one method is used, in this order of precedence:

FieldMethod
CredsFilenats.UserCredentials(CredsFile)
NKeySeednats.NkeyOptionFromSeed(NKeySeed) - an invalid seed is silently ignored and the connection proceeds unauthenticated
Tokennats.Token(Token)
User + Passwordnats.UserInfo(User, Password)

YAML Configuration

Pools can also come from a configuration file. The list-based shape below is the orm.Config struct (yaml tags) consumed by registry.InitByConfig - see Registry. The top-level key is natsPools:

natsPools:
  - code: default
    urls: ["nats://localhost:4222"]
    clientID: my-service
    maxReconnects: -1
    reconnectWaitMs: 1000
    reconnectBufSize: 8388608
    connectTimeoutMs: 5000
    retryOnFailedConnect: true
    authUser: app
    authPassword: secret
    ignoredSubjects: ["legacy.>"]
    streams:
      - name: ORDERS_RAW
        subjects: ["orders.>"]
        maxAgeMs: 604800000
        maxBytes: 1073741824
        maxMsgSize: 1048576
        replicas: 1
        duplicateWindowMs: 600000
    consumers:
      - name: orders-audit
        filterSubjects: ["orders.>"]
        ackWaitMs: 30000
        maxAckPending: 16
        maxDeliver: -1

Per-pool keys: code (required), urls (required), clientID, maxReconnects, reconnectWaitMs, reconnectBufSize, connectTimeoutMs, retryOnFailedConnect, authToken, authUser, authPassword, authCredsFile, authNKeySeed, ignoredSubjects, ignoredConsumers, streams, consumers.

streams entries: name (required), subjects (required), maxAgeMs, maxBytes, maxMsgSize, replicas, duplicateWindowMs, storage, retention. Each becomes a NewNatsStream(name, code) builder registered with RegisterNatsStream.

consumers entries: name (required), filterSubjects, ackWaitMs, maxAckPending, maxDeliver. Each becomes a NewNatsConsumer(name, code) builder registered with RegisterNatsConsumer.

What the YAML loader does not do

  • storage and retention are parsed into the config struct but never applied to the stream builder. Streams from YAML always use the builder defaults (FileStorage, LimitsPolicy). Use the Go builder if you need another value.
  • There is no YAML key for PingInterval.
  • The nested <pool code>: nats: shape read by InitByYaml supports fewer keys (no connectTimeoutMs, retryOnFailedConnect, authNKeySeed, storage, retention, and streams there need an explicit poolCode) - see Registry.
  • ignoredConsumers is stored but unused (see the options table above).

Publishing

The Nats interface is what engine.Nats(code) returns:

type Nats interface {
    GetCode() string
    GetURLs() []string
    GetPoolOptions() *NatsPoolOptions
    Ping() error
    Publish(ctx Context, msg *NatsMessage) error
    PublishWithAck(ctx Context, msg *NatsMessage) (NatsPubAck, error)
    PublishBatch(ctx Context, msgs []*NatsMessage) error
    PublishAsync(ctx Context, msg *NatsMessage, callback func(*NatsMessage, error))
    Consumer(name string) (NatsConsumer, error)
    MustConsumer(name string) NatsConsumer
    ConsumerNames() []string
    GetJetStream() (jetstream.JetStream, error)
    GetConn() (*nats.Conn, error)
    Close()
}

Messages are NatsMessage values:

type NatsMessage struct {
    Subject    string
    Data       []byte
    Headers    nats.Header
    Sequence   uint64    // stream sequence, set on fetched messages
    Timestamp  time.Time // broker timestamp, set on fetched messages
    Deliveries uint64    // how many times JetStream delivered it (starts at 1); 0 on a message you build
}

func NewNatsMessage(subject string) *NatsMessage // Headers is pre-initialised
MethodBehaviour
Publish(ctx, msg)PublishMsg and waits for the JetStream ack.
PublishWithAck(ctx, msg)Same, and returns NatsPubAck{Stream string, Sequence uint64, Duplicate bool}. Duplicate is true when the stream had already seen the message's Nats-Msg-Id header inside its duplicate window and collapsed it - the message will never be delivered.
PublishBatch(ctx, msgs)Publishes every message asynchronously, then waits once for all acks. Wire order is preserved, so stream sequences match a Publish loop. Returns the first error. An empty slice is a no-op.
PublishAsync(ctx, msg, callback)Returns immediately. callback (may be nil) runs in a goroutine with (msg, ackErr), or with (nil, err) if the publish could not even be started.
ctx := engine.NewContext(context.Background())
pool := engine.Nats("events")

msg := orm.NewNatsMessage("orders.created")
msg.Data = []byte(`{"id":1}`)
msg.Headers.Set("Nats-Msg-Id", "order-created-1") // optional JetStream deduplication

ack, err := pool.PublishWithAck(ctx, msg)
if err != nil {
    panic(err)
}
if ack.Duplicate {
    fmt.Println("collapsed by the stream's duplicate window")
}

Warning

A message published to a subject that no stream captures is dropped by the server without an error at the publisher. Make sure a stream covering the subject exists (see Topology alters).

Streams and Consumers

Droplet ORM manages streams of its own (CDC, ORM_ENTITY and ORM_TASK, see below). Any other stream or durable consumer you need is declared with the builders and reconciled onto the broker by GetNatsAlters.

NatsStreamBuilder

registry.RegisterNatsStream(
    orm.NewNatsStream("ORDERS_RAW", "events").
        Subjects("orders.>").
        MaxAge(7 * 24 * time.Hour).
        Duplicates(10 * time.Minute),
)
MethodDefaultDescription
NewNatsStream(streamName, poolCode string)Constructor.
Subjects(subjects ...string)requiredSubjects the stream captures.
Retention(r jetstream.RetentionPolicy)jetstream.LimitsPolicy
Storage(s jetstream.StorageType)jetstream.FileStorage
MaxAge(d time.Duration)unlimitedApplied when > 0.
MaxBytes(n int64)unlimited (-1)Applied when > 0.
MaxMsgSize(n int32)unlimited (-1)Applied when > 0.
Replicas(n int)1
Duplicates(d time.Duration)server defaultDeduplication window for Nats-Msg-Id. Applied when > 0.

Validate() fails with nats stream name is required, nats pool code is required for stream '%s', nats stream '%s' must have at least one subject, nats pool '%s' not registered for stream '%s' or duplicate nats stream '%s' in pool '%s' (already registered in pool '%s').

NatsConsumerBuilder

registry.RegisterNatsConsumer(
    orm.NewNatsConsumer("orders-audit", "events").
        Stream("ORDERS_RAW").
        FilterSubjects("orders.>").
        AckWait(time.Minute).
        MaxAckPending(16),
)
MethodDefaultDescription
NewNatsConsumer(name, poolCode string)name is the durable name.
Stream(name string)guessedBinds the consumer to a stream, for the reconciler and for Fetch. Without it the reconciler picks the first existing stream whose subjects cover the first filter subject (exact match or a .> prefix match), and Fetch scans every stream for a durable of that name. Always set it: durable names are unique per stream, not per broker.
FilterSubjects(subjects ...string)noneOne subject becomes FilterSubject, several become FilterSubjects.
AckWait(d time.Duration)30s
MaxAckPending(n int)1
MaxDeliver(n int)-1 (unlimited)Applied when not 0.
DeliverPolicy(p jetstream.DeliverPolicy)jetstream.DeliverAllPolicy

The ack policy is always AckExplicitPolicy. Validate() fails with nats consumer name is required, nats pool code is required for consumer '%s', nats pool '%s' not registered for consumer '%s' or duplicate nats consumer '%s' in pool '%s' (already registered in pool '%s').

Fetching messages

A registered consumer is obtained from the pool by name and drained with Fetch:

type NatsConsumer interface {
    GetName() string
    GetSettings() *NatsConsumerSettings
    Fetch(ctx Context, batch int, maxWait time.Duration) NatsBatch
    Close()
}
consumer, err := engine.Nats("events").Consumer("orders-audit")
if err != nil {
    panic(err) // "nats pool 'events': consumer 'orders-audit' not registered"
}

for {
    batch := consumer.Fetch(ctx, 50, 2*time.Second)
    if err := batch.Error(); err != nil {
        log.Printf("fetch: %v", err)
        continue
    }
    for _, msg := range batch.Records() {
        if err := handle(msg); err != nil {
            _ = msg.NakWithDelay(10 * time.Second) // redeliver later
            continue
        }
        _ = msg.Ack()
    }
}
  • Consumer(name) knows the durables registered with RegisterNatsConsumer and the ones declared through ConsumerDef. MustConsumer panics instead of returning the error; ConsumerNames() lists them (unsorted).
  • GetSettings() returns the registered definition as *NatsConsumerSettings: Name, Stream, FilterSubjects []string, AckWait, MaxAckPending, MaxDeliver and DeliverPolicy (a ConsumerDef durable leaves DeliverPolicy at its zero value, DeliverAllPolicy).
  • On the first Fetch the durable is resolved on its declared stream (NatsConsumerSettings.Stream, set by NatsConsumerBuilder.Stream, by every ConsumerDef and for the CDC reader). A missing stream or durable is the batch error nats pool '%s': durable consumer '%s' on stream '%s': <cause>. Run GetNatsAlters first.
  • Only a consumer registered without a stream falls back to scanning every stream on the server and binds to the first one that has a durable of that name; if none has it the batch error is nats pool '%s': durable consumer '%s' not found in any stream. When two streams hold a durable with the same name - typically after a stream was renamed and the old one kept - that scan can pick the wrong one, which is why Stream should always be set. Consumers from YAML have no stream key and always use the scan.
  • The resolved durable is cached and resolved again only when the pool replaced its connection (see Connection lifecycle).
  • A fetch that times out with no messages is an idle poll, not an error: batch.Error() is nil and batch.IsEmpty() is true.
  • NatsBatch offers Records() []*NatsMessage, EachRecord(func(*NatsMessage)), EachError(func(error)), Error() error and IsEmpty() bool.
  • Fetched messages carry Sequence, Timestamp and Deliveries from the JetStream metadata.
  • msg.Ack(), msg.Nak(), msg.NakWithDelay(d) and msg.Term() act on the JetStream message. On a message that was not fetched (one you built yourself) they are no-ops returning nil.
  • NatsConsumer.Close() is a no-op; close the pool instead.

Tips

Plain Nak() redelivers immediately, which turns a permanently failing handler into a hot loop. Use NakWithDelay for a backoff.

Streams Managed by Droplet ORM

When you use entity events or tasks, GetNatsAlters also takes care of these streams, on the pool selected by CDCStreamOptions.NatsPool / EntityStreamOptions.NatsPool / TaskStreamOptions.NatsPool:

StreamConstantSubjectsCarriesCreated when
CDCorm.CDCStreamNamecdc.>the change feed: one Debezium envelope per row change on cdc.<database>.<table>at least one generated cdc entity is registered
ORM_ENTITYorm.EntityStreamNameorm.entity.>, orm.replay.>changes forwarded by the CDC reader, and replaysat least one ConsumerDef is registered
ORM_TASKorm.TaskStreamNameorm.task.>dispatched tasksat least one task is registered

ORM_ENTITY and ORM_TASK are tuned with EntityStreamOptions / TaskStreamOptions (fields Storage, Replicas, MaxAge, MaxBytes, DuplicateWindow, NatsPool; defaults: pool default, Replicas=1, MaxAge=7d, DuplicateWindow=10m, MaxBytes unbounded, Storage file) and are fully owned by Droplet ORM: created, updated when the options change. See Entity events and Tasks.

CDC is different, because a Debezium deployment may write to it and own its limits: it is created when missing from CDCStreamOptions (DuplicateWindow=2m, otherwise the same defaults) and after that never updated and never deleted. See The change feed.

Durable consumers Droplet ORM declares itself:

DurableStreamFilter subjects
orm-cdc-reader (orm.CDCReaderName)CDCcdc.<database>.<table> of every cdc entity
<ConsumerDef.Name> of an entity consumerORM_ENTITYorm.entity.<table> per declared entity, orm.replay.<name>.>
<ConsumerDef.Name> of a task consumerORM_TASKorm.task.<queue>.> per declared queue

Subjects are typed as orm.Subject and built with:

FunctionResult
CDCSubject(database, table string) Subjectcdc.<database>.<table>
EntitySubject(table string) Subjectorm.entity.<table>
ReplaySubject(consumer ConsumerName, table string) Subjectorm.replay.<consumer>.<table>
TaskSubject(queue Queue, task TaskName) Subjectorm.task.<queue>.<TaskName>

Related string types: ConsumerName (also the JetStream durable name), Queue, TaskName, and the constants DefaultQueue Queue = "default" and CDCReaderName ConsumerName = "orm-cdc-reader".

Task messages carry the header Orm-Job-Run-ID (orm.HeaderJobRunID). Change-feed and entity messages carry no ORM header besides Nats-Msg-Id; the operation is the envelope's op field.

These names are wire names

Stream names, subjects, durable names and the header are what other processes and the broker see. They changed with the rename from fluxaorm (FLUXA_ENTITY, FLUXA_TASK, fluxa.*, Fluxa-Job-Run-ID); streams with the old names are left alone by GetNatsAlters and nothing reads them any more. See Migrating from fluxaorm for the cutover.

Topology Alters

GetNatsAlters compares the desired topology with the broker and returns the operations needed to synchronise them. It is the NATS counterpart of GetAlters for MySQL - the two are separate, run both:

func GetNatsAlters(ctx Context) ([]NatsAlter, error)

type NatsAlter struct {
    Description string      // e.g. "CREATE nats stream 'ORM_ENTITY'"
    PoolCode    string
    Kind        AlterKind   // AlterKindCreateTable, AlterKindModifyTable or AlterKindDropTable
    Safety      AlterSafety // AlterSafe or AlterDestructive
}

func (a NatsAlter) IsSafe() bool
func (a NatsAlter) Exec(ctx Context) error
ctx := engine.NewContext(context.Background())

// MySQL schema first (tables), then the JetStream topology.
alters, err := orm.GetAlters(ctx)
if err != nil {
    panic(err)
}
for _, alter := range alters {
    if err := alter.Exec(ctx); err != nil {
        panic(err)
    }
}

natsAlters, err := orm.GetNatsAlters(ctx)
if err != nil {
    panic(err)
}
for _, alter := range natsAlters {
    if !alter.IsSafe() {
        // Stream deletions and subject-narrowing updates break the previous
        // code version if it is still running. Gate them behind a manual step.
        fmt.Printf("skipping destructive alter: %s (pool %s)\n", alter.Description, alter.PoolCode)
        continue
    }
    if err := alter.Exec(ctx); err != nil {
        panic(err)
    }
}

The desired state is: every RegisterNatsStream builder, plus ORM_ENTITY and ORM_TASK when applicable; every RegisterNatsConsumer builder, plus one durable per ConsumerDef (bound to its stream, filtering all its subjects, with the ConsumerDef's AckWait, MaxAckPending, MaxDeliver) and the orm-cdc-reader durable on CDC when at least one cdc entity is registered. The CDC stream itself is handled separately: created when missing, otherwise untouched.

SituationAlterKindSafety
Desired stream missing on the broker (also CDC)CREATE nats stream '<name>'AlterKindCreateTablesafe
Desired stream exists with a different configUPDATE nats stream '<name>' - pushes the whole desired configAlterKindModifyTablesafe if the desired subject list is a superset of the existing one, destructive otherwise
Stream exists, is not desired, and its name starts with ORM_ or orm_DELETE nats stream '<name>'AlterKindDropTabledestructive
Every desired consumerENSURE nats consumer '<name>' - CreateOrUpdateConsumerAlterKindCreateTablesafe

Details worth knowing:

  • Two stream configs are equal when Retention, Storage, Replicas, MaxAge, MaxBytes, MaxMsgSize, Duplicates and the (sorted) subject lists match.
  • Only streams whose names start with ORM_ or orm_ and that are not registered are deleted. Every other stream is never deleted, whatever else is on the broker - this includes CDC, streams of other applications, and FLUXA_* streams left from before the rename. An ORM_-prefixed stream that carries a subject listed in the pool's IgnoredSubjects is not deleted either.
  • Do not name your own RegisterNatsStream streams with an ORM_/orm_ prefix unless every process sharing the broker registers them: an instance that does not know the stream reports a DELETE for it.
  • An existing CDC stream is never compared with CDCStreamOptions; no UPDATE is ever reported for it.
  • Consumers are never diffed or deleted. A durable left on a stream that is no longer used (for example on an old FLUXA_* stream) stays there until the stream is removed by hand. ENSURE is always reported as safe and updates the durable in place, so widening a consumer's filter list (for example adding an entity to a ConsumerDef) keeps its position in the stream. A change that narrows the filter is applied too and will stop deliveries the previous code version still expects - it is not flagged.
  • GetNatsAlters connects to every pool that has at least one desired stream. A pool that has desired consumers but no desired streams is skipped entirely.
  • The result is sorted by Description, which places CREATE before ENSURE before UPDATE.

Testing

orm.PrepareTablesWithNats and orm.PrepareTablesWithConsumers apply GetNatsAlters for you - see Testing.

Logging and Metrics

NATS operations are logged when a query logger is registered with Nats: true (see Queries log):

ctx.RegisterQueryLogger(myHandler, orm.QueryLoggerOptions{Nats: true})
// or, to stderr:
ctx.EnableQueryDebugCustom(orm.QueryLoggerOptions{Nats: true})

Log source is nats. Operations: PUBLISH (query subject: <subject>), PUBLISH_BATCH (messages: <n>), PUBLISH_ASYNC, FETCH (<n> messages fetched, pool label <pool>/<consumer>), from the consumer runtime CDC_HANDLE / TASK_HANDLE (handler errors, pool label is the consumer name, no duration) and from the CDC reader CDC_READ (dropped undecodable events and ack errors, pool label orm-cdc-reader).

Prometheus metrics (see Metrics):

MetricTypeLabels
orm_nats_operations_secondshistogramoperation (publish, publish_async, fetch), pool, source, consumer (empty for publishes, durable name for fetch)
orm_nats_operations_errorscounterpool, source, consumer
orm_nats_publish_batch_sizehistogrampool, source - observed once per PublishBatch with the batch length
orm_cdc_messages_totalcounterconsumer, entity, op - one per message fetched by a declared consumer or by the CDC reader (consumer="orm-cdc-reader")
orm_stream_consume_lag_secondshistogramconsumer - broker timestamp to consume time

Closing

engine.Nats(code).Close() closes the underlying connection. The next operation on the pool opens a new one (see Connection lifecycle), so call it on shutdown only:

defer engine.Nats(orm.DefaultPoolCode).Close()
Edit this page
Last Updated: 11/16/26, 1:49 PM
Prev
ClickHouse Schema
Next
Entity Change Events (CDC)