NATS and JetStream
Droplet ORM uses NATS JetStream as its messaging transport. Three features are built on top of it:
- Entity change events - every write to an entity tagged
orm:"cdc"goes on theCDCchange feed and is forwarded to consumers. - Consumers - typed, generated handlers that read entity events and tasks.
- Tasks - background jobs dispatched onto queues, with a run history table.
A NATS pool is a connection to one or more NATS servers. This page covers the pool itself, the low-level publish and fetch API, hand-registered streams and consumers, and the topology reconciliation (GetNatsAlters) that creates and updates streams and durable consumers on the broker.
Pool Registration
Register a pool with RegisterNats:
import "github.com/DropletLive/orm"
registry := orm.NewRegistry()
// Pool with default options (nil is allowed and becomes an empty NatsPoolOptions):
registry.RegisterNats([]string{"nats://localhost:4222"}, orm.DefaultPoolCode, nil)
// Pool with several servers and options:
registry.RegisterNats([]string{"nats://nats-1:4222", "nats://nats-2:4222"}, "events", &orm.NatsPoolOptions{
ClientID: "my-service",
RetryOnFailedConnect: true,
Auth: &orm.NatsAuthConfig{
User: "app",
Password: "secret",
},
})
RegisterNats(urls []string, poolCode string, options *NatsPoolOptions)
The pool is available from the engine as engine.Nats(code) (returns nil for an unknown code) and all pools as engine.Registry().NatsPools().
Lazy connection
registry.Validate() only builds the pool object; nothing connects to NATS until the first operation that needs a connection (Publish*, Consumer, GetJetStream, GetConn, Ping). Ping() forces a connection attempt and is the way to fail fast at startup.
The features built on NATS pick their pool with the NatsPool field of EntityStreamOptions, CDCStreamOptions, TaskStreamOptions and ConsumerDef. They default to orm.DefaultPoolCode ("default") - CDCStreamOptions.NatsPool to the entity stream's pool - so registering the pool under the default code needs no further wiring.
Connection lifecycle
A pool owns one connection and one JetStream context, shared by every publish and every consumer made from it.
- A reconnecting connection is kept. When the server goes away the NATS client reconnects by itself (
MaxReconnects,ReconnectWait). The pool hands out the same connection the whole time; operations issued while it is down fail or wait according to the client's own rules, and work again when it is back. Nothing made from the pool has to be recreated. - A closed connection is replaced. If the connection is closed - the reconnect attempts ran out with a finite
MaxReconnects, orClose()was called - the next operation dials a new one. - Consumers follow the pool. A
NatsConsumer(and everyStreamConsumerbuilt on one) asks the pool for the current JetStream context on eachFetchand re-resolves its durable when the context changed. A consumer obtained once at startup keeps working across reconnects and across a replaced connection; a fetch loop only has to log the error, back off and callFetch/Consumeagain. GetConn()andGetJetStream()return the current connection. Do not cache them across aClose(), and do not close the returned*nats.Connyourself - callpool.Close().
NatsPoolOptions
| Field | Type | Behaviour |
|---|---|---|
ClientID | string | Passed as the connection name (nats.Name). |
MaxReconnects | int | Default -1 (reconnect forever). Applied only when not 0. |
ReconnectWait | time.Duration | Default 1s. Applied when > 0. |
ReconnectBufSize | int | Applied when > 0. |
PingInterval | time.Duration | Applied when > 0. |
ConnectTimeout | time.Duration | Applied when > 0 (nats.Timeout). |
RetryOnFailedConnect | bool | When true the client keeps retrying if the initial connect fails. |
Auth | *NatsAuthConfig | Authentication, see below. |
IgnoredSubjects | []string | Existing ORM_*/orm_* streams carrying any of these subjects are never deleted by GetNatsAlters. The list is only taken into account when at least one stream is registered with RegisterNatsStream. |
IgnoredConsumers | []string | Parsed and stored, but not read anywhere - GetNatsAlters never deletes consumers, so there is nothing to exempt. Setting it has no effect today. |
NatsAuthConfig
Exactly one method is used, in this order of precedence:
| Field | Method |
|---|---|
CredsFile | nats.UserCredentials(CredsFile) |
NKeySeed | nats.NkeyOptionFromSeed(NKeySeed) - an invalid seed is silently ignored and the connection proceeds unauthenticated |
Token | nats.Token(Token) |
User + Password | nats.UserInfo(User, Password) |
YAML Configuration
Pools can also come from a configuration file. The list-based shape below is the orm.Config struct (yaml tags) consumed by registry.InitByConfig - see Registry. The top-level key is natsPools:
natsPools:
- code: default
urls: ["nats://localhost:4222"]
clientID: my-service
maxReconnects: -1
reconnectWaitMs: 1000
reconnectBufSize: 8388608
connectTimeoutMs: 5000
retryOnFailedConnect: true
authUser: app
authPassword: secret
ignoredSubjects: ["legacy.>"]
streams:
- name: ORDERS_RAW
subjects: ["orders.>"]
maxAgeMs: 604800000
maxBytes: 1073741824
maxMsgSize: 1048576
replicas: 1
duplicateWindowMs: 600000
consumers:
- name: orders-audit
filterSubjects: ["orders.>"]
ackWaitMs: 30000
maxAckPending: 16
maxDeliver: -1
Per-pool keys: code (required), urls (required), clientID, maxReconnects, reconnectWaitMs, reconnectBufSize, connectTimeoutMs, retryOnFailedConnect, authToken, authUser, authPassword, authCredsFile, authNKeySeed, ignoredSubjects, ignoredConsumers, streams, consumers.
streams entries: name (required), subjects (required), maxAgeMs, maxBytes, maxMsgSize, replicas, duplicateWindowMs, storage, retention. Each becomes a NewNatsStream(name, code) builder registered with RegisterNatsStream.
consumers entries: name (required), filterSubjects, ackWaitMs, maxAckPending, maxDeliver. Each becomes a NewNatsConsumer(name, code) builder registered with RegisterNatsConsumer.
What the YAML loader does not do
storageandretentionare parsed into the config struct but never applied to the stream builder. Streams from YAML always use the builder defaults (FileStorage,LimitsPolicy). Use the Go builder if you need another value.- There is no YAML key for
PingInterval. - The nested
<pool code>: nats:shape read byInitByYamlsupports fewer keys (noconnectTimeoutMs,retryOnFailedConnect,authNKeySeed,storage,retention, and streams there need an explicitpoolCode) - see Registry. ignoredConsumersis stored but unused (see the options table above).
Publishing
The Nats interface is what engine.Nats(code) returns:
type Nats interface {
GetCode() string
GetURLs() []string
GetPoolOptions() *NatsPoolOptions
Ping() error
Publish(ctx Context, msg *NatsMessage) error
PublishWithAck(ctx Context, msg *NatsMessage) (NatsPubAck, error)
PublishBatch(ctx Context, msgs []*NatsMessage) error
PublishAsync(ctx Context, msg *NatsMessage, callback func(*NatsMessage, error))
Consumer(name string) (NatsConsumer, error)
MustConsumer(name string) NatsConsumer
ConsumerNames() []string
GetJetStream() (jetstream.JetStream, error)
GetConn() (*nats.Conn, error)
Close()
}
Messages are NatsMessage values:
type NatsMessage struct {
Subject string
Data []byte
Headers nats.Header
Sequence uint64 // stream sequence, set on fetched messages
Timestamp time.Time // broker timestamp, set on fetched messages
Deliveries uint64 // how many times JetStream delivered it (starts at 1); 0 on a message you build
}
func NewNatsMessage(subject string) *NatsMessage // Headers is pre-initialised
| Method | Behaviour |
|---|---|
Publish(ctx, msg) | PublishMsg and waits for the JetStream ack. |
PublishWithAck(ctx, msg) | Same, and returns NatsPubAck{Stream string, Sequence uint64, Duplicate bool}. Duplicate is true when the stream had already seen the message's Nats-Msg-Id header inside its duplicate window and collapsed it - the message will never be delivered. |
PublishBatch(ctx, msgs) | Publishes every message asynchronously, then waits once for all acks. Wire order is preserved, so stream sequences match a Publish loop. Returns the first error. An empty slice is a no-op. |
PublishAsync(ctx, msg, callback) | Returns immediately. callback (may be nil) runs in a goroutine with (msg, ackErr), or with (nil, err) if the publish could not even be started. |
ctx := engine.NewContext(context.Background())
pool := engine.Nats("events")
msg := orm.NewNatsMessage("orders.created")
msg.Data = []byte(`{"id":1}`)
msg.Headers.Set("Nats-Msg-Id", "order-created-1") // optional JetStream deduplication
ack, err := pool.PublishWithAck(ctx, msg)
if err != nil {
panic(err)
}
if ack.Duplicate {
fmt.Println("collapsed by the stream's duplicate window")
}
Warning
A message published to a subject that no stream captures is dropped by the server without an error at the publisher. Make sure a stream covering the subject exists (see Topology alters).
Streams and Consumers
Droplet ORM manages streams of its own (CDC, ORM_ENTITY and ORM_TASK, see below). Any other stream or durable consumer you need is declared with the builders and reconciled onto the broker by GetNatsAlters.
NatsStreamBuilder
registry.RegisterNatsStream(
orm.NewNatsStream("ORDERS_RAW", "events").
Subjects("orders.>").
MaxAge(7 * 24 * time.Hour).
Duplicates(10 * time.Minute),
)
| Method | Default | Description |
|---|---|---|
NewNatsStream(streamName, poolCode string) | Constructor. | |
Subjects(subjects ...string) | required | Subjects the stream captures. |
Retention(r jetstream.RetentionPolicy) | jetstream.LimitsPolicy | |
Storage(s jetstream.StorageType) | jetstream.FileStorage | |
MaxAge(d time.Duration) | unlimited | Applied when > 0. |
MaxBytes(n int64) | unlimited (-1) | Applied when > 0. |
MaxMsgSize(n int32) | unlimited (-1) | Applied when > 0. |
Replicas(n int) | 1 | |
Duplicates(d time.Duration) | server default | Deduplication window for Nats-Msg-Id. Applied when > 0. |
Validate() fails with nats stream name is required, nats pool code is required for stream '%s', nats stream '%s' must have at least one subject, nats pool '%s' not registered for stream '%s' or duplicate nats stream '%s' in pool '%s' (already registered in pool '%s').
NatsConsumerBuilder
registry.RegisterNatsConsumer(
orm.NewNatsConsumer("orders-audit", "events").
Stream("ORDERS_RAW").
FilterSubjects("orders.>").
AckWait(time.Minute).
MaxAckPending(16),
)
| Method | Default | Description |
|---|---|---|
NewNatsConsumer(name, poolCode string) | name is the durable name. | |
Stream(name string) | guessed | Binds the consumer to a stream, for the reconciler and for Fetch. Without it the reconciler picks the first existing stream whose subjects cover the first filter subject (exact match or a .> prefix match), and Fetch scans every stream for a durable of that name. Always set it: durable names are unique per stream, not per broker. |
FilterSubjects(subjects ...string) | none | One subject becomes FilterSubject, several become FilterSubjects. |
AckWait(d time.Duration) | 30s | |
MaxAckPending(n int) | 1 | |
MaxDeliver(n int) | -1 (unlimited) | Applied when not 0. |
DeliverPolicy(p jetstream.DeliverPolicy) | jetstream.DeliverAllPolicy |
The ack policy is always AckExplicitPolicy. Validate() fails with nats consumer name is required, nats pool code is required for consumer '%s', nats pool '%s' not registered for consumer '%s' or duplicate nats consumer '%s' in pool '%s' (already registered in pool '%s').
Fetching messages
A registered consumer is obtained from the pool by name and drained with Fetch:
type NatsConsumer interface {
GetName() string
GetSettings() *NatsConsumerSettings
Fetch(ctx Context, batch int, maxWait time.Duration) NatsBatch
Close()
}
consumer, err := engine.Nats("events").Consumer("orders-audit")
if err != nil {
panic(err) // "nats pool 'events': consumer 'orders-audit' not registered"
}
for {
batch := consumer.Fetch(ctx, 50, 2*time.Second)
if err := batch.Error(); err != nil {
log.Printf("fetch: %v", err)
continue
}
for _, msg := range batch.Records() {
if err := handle(msg); err != nil {
_ = msg.NakWithDelay(10 * time.Second) // redeliver later
continue
}
_ = msg.Ack()
}
}
Consumer(name)knows the durables registered withRegisterNatsConsumerand the ones declared throughConsumerDef.MustConsumerpanics instead of returning the error;ConsumerNames()lists them (unsorted).GetSettings()returns the registered definition as*NatsConsumerSettings:Name,Stream,FilterSubjects []string,AckWait,MaxAckPending,MaxDeliverandDeliverPolicy(aConsumerDefdurable leavesDeliverPolicyat its zero value,DeliverAllPolicy).- On the first
Fetchthe durable is resolved on its declared stream (NatsConsumerSettings.Stream, set byNatsConsumerBuilder.Stream, by everyConsumerDefand for the CDC reader). A missing stream or durable is the batch errornats pool '%s': durable consumer '%s' on stream '%s': <cause>. RunGetNatsAltersfirst. - Only a consumer registered without a stream falls back to scanning every stream on the server and binds to the first one that has a durable of that name; if none has it the batch error is
nats pool '%s': durable consumer '%s' not found in any stream. When two streams hold a durable with the same name - typically after a stream was renamed and the old one kept - that scan can pick the wrong one, which is whyStreamshould always be set. Consumers from YAML have nostreamkey and always use the scan. - The resolved durable is cached and resolved again only when the pool replaced its connection (see Connection lifecycle).
- A fetch that times out with no messages is an idle poll, not an error:
batch.Error()isnilandbatch.IsEmpty()istrue. NatsBatchoffersRecords() []*NatsMessage,EachRecord(func(*NatsMessage)),EachError(func(error)),Error() errorandIsEmpty() bool.- Fetched messages carry
Sequence,TimestampandDeliveriesfrom the JetStream metadata. msg.Ack(),msg.Nak(),msg.NakWithDelay(d)andmsg.Term()act on the JetStream message. On a message that was not fetched (one you built yourself) they are no-ops returningnil.NatsConsumer.Close()is a no-op; close the pool instead.
Tips
Plain Nak() redelivers immediately, which turns a permanently failing handler into a hot loop. Use NakWithDelay for a backoff.
Streams Managed by Droplet ORM
When you use entity events or tasks, GetNatsAlters also takes care of these streams, on the pool selected by CDCStreamOptions.NatsPool / EntityStreamOptions.NatsPool / TaskStreamOptions.NatsPool:
| Stream | Constant | Subjects | Carries | Created when |
|---|---|---|---|---|
CDC | orm.CDCStreamName | cdc.> | the change feed: one Debezium envelope per row change on cdc.<database>.<table> | at least one generated cdc entity is registered |
ORM_ENTITY | orm.EntityStreamName | orm.entity.>, orm.replay.> | changes forwarded by the CDC reader, and replays | at least one ConsumerDef is registered |
ORM_TASK | orm.TaskStreamName | orm.task.> | dispatched tasks | at least one task is registered |
ORM_ENTITY and ORM_TASK are tuned with EntityStreamOptions / TaskStreamOptions (fields Storage, Replicas, MaxAge, MaxBytes, DuplicateWindow, NatsPool; defaults: pool default, Replicas=1, MaxAge=7d, DuplicateWindow=10m, MaxBytes unbounded, Storage file) and are fully owned by Droplet ORM: created, updated when the options change. See Entity events and Tasks.
CDC is different, because a Debezium deployment may write to it and own its limits: it is created when missing from CDCStreamOptions (DuplicateWindow=2m, otherwise the same defaults) and after that never updated and never deleted. See The change feed.
Durable consumers Droplet ORM declares itself:
| Durable | Stream | Filter subjects |
|---|---|---|
orm-cdc-reader (orm.CDCReaderName) | CDC | cdc.<database>.<table> of every cdc entity |
<ConsumerDef.Name> of an entity consumer | ORM_ENTITY | orm.entity.<table> per declared entity, orm.replay.<name>.> |
<ConsumerDef.Name> of a task consumer | ORM_TASK | orm.task.<queue>.> per declared queue |
Subjects are typed as orm.Subject and built with:
| Function | Result |
|---|---|
CDCSubject(database, table string) Subject | cdc.<database>.<table> |
EntitySubject(table string) Subject | orm.entity.<table> |
ReplaySubject(consumer ConsumerName, table string) Subject | orm.replay.<consumer>.<table> |
TaskSubject(queue Queue, task TaskName) Subject | orm.task.<queue>.<TaskName> |
Related string types: ConsumerName (also the JetStream durable name), Queue, TaskName, and the constants DefaultQueue Queue = "default" and CDCReaderName ConsumerName = "orm-cdc-reader".
Task messages carry the header Orm-Job-Run-ID (orm.HeaderJobRunID). Change-feed and entity messages carry no ORM header besides Nats-Msg-Id; the operation is the envelope's op field.
These names are wire names
Stream names, subjects, durable names and the header are what other processes and the broker see. They changed with the rename from fluxaorm (FLUXA_ENTITY, FLUXA_TASK, fluxa.*, Fluxa-Job-Run-ID); streams with the old names are left alone by GetNatsAlters and nothing reads them any more. See Migrating from fluxaorm for the cutover.
Topology Alters
GetNatsAlters compares the desired topology with the broker and returns the operations needed to synchronise them. It is the NATS counterpart of GetAlters for MySQL - the two are separate, run both:
func GetNatsAlters(ctx Context) ([]NatsAlter, error)
type NatsAlter struct {
Description string // e.g. "CREATE nats stream 'ORM_ENTITY'"
PoolCode string
Kind AlterKind // AlterKindCreateTable, AlterKindModifyTable or AlterKindDropTable
Safety AlterSafety // AlterSafe or AlterDestructive
}
func (a NatsAlter) IsSafe() bool
func (a NatsAlter) Exec(ctx Context) error
ctx := engine.NewContext(context.Background())
// MySQL schema first (tables), then the JetStream topology.
alters, err := orm.GetAlters(ctx)
if err != nil {
panic(err)
}
for _, alter := range alters {
if err := alter.Exec(ctx); err != nil {
panic(err)
}
}
natsAlters, err := orm.GetNatsAlters(ctx)
if err != nil {
panic(err)
}
for _, alter := range natsAlters {
if !alter.IsSafe() {
// Stream deletions and subject-narrowing updates break the previous
// code version if it is still running. Gate them behind a manual step.
fmt.Printf("skipping destructive alter: %s (pool %s)\n", alter.Description, alter.PoolCode)
continue
}
if err := alter.Exec(ctx); err != nil {
panic(err)
}
}
The desired state is: every RegisterNatsStream builder, plus ORM_ENTITY and ORM_TASK when applicable; every RegisterNatsConsumer builder, plus one durable per ConsumerDef (bound to its stream, filtering all its subjects, with the ConsumerDef's AckWait, MaxAckPending, MaxDeliver) and the orm-cdc-reader durable on CDC when at least one cdc entity is registered. The CDC stream itself is handled separately: created when missing, otherwise untouched.
| Situation | Alter | Kind | Safety |
|---|---|---|---|
Desired stream missing on the broker (also CDC) | CREATE nats stream '<name>' | AlterKindCreateTable | safe |
| Desired stream exists with a different config | UPDATE nats stream '<name>' - pushes the whole desired config | AlterKindModifyTable | safe if the desired subject list is a superset of the existing one, destructive otherwise |
Stream exists, is not desired, and its name starts with ORM_ or orm_ | DELETE nats stream '<name>' | AlterKindDropTable | destructive |
| Every desired consumer | ENSURE nats consumer '<name>' - CreateOrUpdateConsumer | AlterKindCreateTable | safe |
Details worth knowing:
- Two stream configs are equal when
Retention,Storage,Replicas,MaxAge,MaxBytes,MaxMsgSize,Duplicatesand the (sorted) subject lists match. - Only streams whose names start with
ORM_ororm_and that are not registered are deleted. Every other stream is never deleted, whatever else is on the broker - this includesCDC, streams of other applications, andFLUXA_*streams left from before the rename. AnORM_-prefixed stream that carries a subject listed in the pool'sIgnoredSubjectsis not deleted either. - Do not name your own
RegisterNatsStreamstreams with anORM_/orm_prefix unless every process sharing the broker registers them: an instance that does not know the stream reports aDELETEfor it. - An existing
CDCstream is never compared withCDCStreamOptions; noUPDATEis ever reported for it. - Consumers are never diffed or deleted. A durable left on a stream that is no longer used (for example on an old
FLUXA_*stream) stays there until the stream is removed by hand.ENSUREis always reported as safe and updates the durable in place, so widening a consumer's filter list (for example adding an entity to aConsumerDef) keeps its position in the stream. A change that narrows the filter is applied too and will stop deliveries the previous code version still expects - it is not flagged. GetNatsAltersconnects to every pool that has at least one desired stream. A pool that has desired consumers but no desired streams is skipped entirely.- The result is sorted by
Description, which placesCREATEbeforeENSUREbeforeUPDATE.
Testing
orm.PrepareTablesWithNats and orm.PrepareTablesWithConsumers apply GetNatsAlters for you - see Testing.
Logging and Metrics
NATS operations are logged when a query logger is registered with Nats: true (see Queries log):
ctx.RegisterQueryLogger(myHandler, orm.QueryLoggerOptions{Nats: true})
// or, to stderr:
ctx.EnableQueryDebugCustom(orm.QueryLoggerOptions{Nats: true})
Log source is nats. Operations: PUBLISH (query subject: <subject>), PUBLISH_BATCH (messages: <n>), PUBLISH_ASYNC, FETCH (<n> messages fetched, pool label <pool>/<consumer>), from the consumer runtime CDC_HANDLE / TASK_HANDLE (handler errors, pool label is the consumer name, no duration) and from the CDC reader CDC_READ (dropped undecodable events and ack errors, pool label orm-cdc-reader).
Prometheus metrics (see Metrics):
| Metric | Type | Labels |
|---|---|---|
orm_nats_operations_seconds | histogram | operation (publish, publish_async, fetch), pool, source, consumer (empty for publishes, durable name for fetch) |
orm_nats_operations_errors | counter | pool, source, consumer |
orm_nats_publish_batch_size | histogram | pool, source - observed once per PublishBatch with the batch length |
orm_cdc_messages_total | counter | consumer, entity, op - one per message fetched by a declared consumer or by the CDC reader (consumer="orm-cdc-reader") |
orm_stream_consume_lag_seconds | histogram | consumer - broker timestamp to consume time |
Closing
engine.Nats(code).Close() closes the underlying connection. The next operation on the pool opens a new one (see Connection lifecycle), so call it on shutdown only:
defer engine.Nats(orm.DefaultPoolCode).Close()