Droplet ORM
Guide
GitHub
Guide
GitHub
    • Introduction
    • Registry
    • Data Pools
    • Entities
    • Entity Fields
    • MySQL Indexes
    • Code Generation
    • Engine
    • Context
    • Schema Update
    • CRUD Operations
    • Transactions
    • Search
    • Elasticsearch
    • MySQL Queries
    • ClickHouse Queries
    • ClickHouse Schema
    • NATS and JetStream
    • Entity Change Events (CDC)
    • Consumers
    • Tasks and Job Runs
    • Transactional Outbox
    • Context Cache
    • Redis Cache
    • Fake Delete
    • Lifecycle Callbacks
    • Metrics
    • Redis Operations
    • Distributed Lock
    • Queries Log
    • Testing
    • Migrating from fluxaorm

Code Generation

Droplet ORM uses no reflection at runtime. You describe entities as plain Go structs (see Entities), register them in a Registry, and run orm.Generate(). The generator emits one Go file per entity containing a typed Provider (loading, searching, factory and metadata methods) and a typed Entity (getters, setters, dirty tracking and the SQL it needs to persist itself), plus a shared enums package. This page is a reference for what is generated and how to call it.

Running the generator

func Generate(engine Engine, outputDirectory string) error

Generate needs a validated Engine. Use registry.ValidateForCodeGen() rather than Validate(): it builds the entity schemas without opening a single connection to MySQL, Redis or NATS, so the generator can run on a machine that has no access to the databases (a developer laptop, CI). Pools still have to be registered so that tags such as orm:"mysql=logs" or orm:"redisCache" can be resolved, but the addresses are never dialled.

Put the call in a small program and run it whenever an entity struct changes:

// cmd/generate/main.go
package main

import (
    "os"

    "github.com/DropletLive/orm"

    "myapp/model"
)

func main() {
    registry := orm.NewRegistry()
    registry.RegisterMySQL("user:password@tcp(localhost:3306)/db", orm.DefaultPoolCode, &orm.MySQLOptions{})
    registry.RegisterRedis("localhost:6379", 0, orm.DefaultPoolCode, nil)
    registry.RegisterEntity(model.CategoryEntity{}, model.UserEntity{})

    engine, err := registry.ValidateForCodeGen()
    if err != nil {
        panic(err)
    }
    if err = os.MkdirAll("entities", 0o755); err != nil {
        panic(err)
    }
    if err = orm.Generate(engine, "entities"); err != nil {
        panic(err)
    }
}
go run ./cmd/generate

What Generate does, in order:

  1. Checks the output directory: it must be non-empty (output directory is empty), exist (output directory does not exist: <path>), be a directory and be writable.
  2. Walks up from the output directory until it finds a go.mod and reads its module line. The import path of the generated enums package is <module>/<relative path of the output dir>/enums, so the output directory must live inside a Go module.
  3. Deletes every regular file in the output directory, whatever its extension. Sub-directories (such as enums/) are kept: the file of every enum still in use is rewritten, but a file left over from an enum that no longer exists is not removed. Never put hand-written files in the output directory.
  4. Writes one file per registered entity, then providers.go, then the consumer files when consumers or tasks are registered.
  5. Deletes every regular file in <outputDirectory>/esfields/ and, when Elasticsearch documents are registered, writes one file per document there.
  6. Runs every file through go/format.

The package name of the generated code is the base name of the output directory (entities in the example above). Commit the generated files together with the entity structs that produced them.

Upgrading Droplet ORM

Regenerate all entities after updating the ORM dependency, even when your model structs have not changed. The generator and runtime must agree on the entity persistence protocol.

The transaction save fix requires generated snapshot and rollback support, including PrivateSnapshot, PrivateRollback and PrivateIsSnapshot. Older generated entities can still compile, but Save returns orm.ErrEntityNeedsRegeneration rather than using an entity without this support. Run your generator with the updated dependency, then rebuild and test the application. Save, Transaction and the public setters keep their existing signatures.

The change feed requires regenerated code as well: <Entity>DirtyEvent is now orm.DirtyEvent[<Entity>] with read-only entities in Before/After, and the generated init() calls orm.RegisterCDCEntity. Code generated for the previous event format does not compile against the current runtime; handlers that read map[string]any snapshots must be rewritten to use getters (see Entity Events).

The GetByID and GetByIDs allocation improvements are emitted into generated providers. Rerun your application generator with the updated ORM dependency to adopt them; changing the dependency alone does not rewrite existing providers. These optimizations preserve method signatures, cache behavior, and missing-row behavior, as well as input order and duplicate handling in GetByIDs. Newly generated readers require the updated ORM runtime for numeric scan adapters and Redis command batching; database/sql conversions, NULL handling, overflow errors, and the Redis row-cache format are preserved.

Output layout

FileContent
<TableName>.goOne file per entity. The file name is the table name verbatim: UserEntity.go for a struct UserEntity, users.go when the struct carries orm:"table=users".
providers.govar AllProviders = []orm.EntityProvider{...}, pointers to every provider sorted by table name.
consumers.go, <consumer>_consumer.goOnly when consumers or tasks are registered. See Consumers and Tasks.
enums/<EnumName>.goPackage enums, one file per enum name (see Enums).
esfields/<document_name>.goPackage esfields, one file per registered Elasticsearch document (see Elasticsearch fields).

Naming rules

  • The table name is the struct type name unless the ID field carries orm:"table=...".
  • The entity type is the table name with its first letter capitalised and underscores removed, capitalising each segment: UserEntity stays UserEntity, a table user_accounts becomes UserAccounts.
  • The provider variable is <Entity>Provider (UserEntityProvider). Its type, the fields struct and the SQL row struct are unexported (userEntityProvider, userEntityFields, userEntitySQLRow).
  • The cache index stored in the provider is the fully qualified Go type name of the source struct (model.UserEntity). It is assigned at validation time and used as the key of the context cache and of the lifecycle handlers.

The examples on this and the following pages use these two structs, defined in a package model and generated into a package entities:

package model

import (
    "time"

    "github.com/DropletLive/orm"
)

type CategoryEntity struct {
    ID   uint64 `orm:"redisCache"`
    Code string `orm:"required;length=10"`
    Name string `orm:"required;length=100"`
}

func (e CategoryEntity) UniqueIndexes() [][]string {
    return [][]string{{"Code"}}
}

func (e CategoryEntity) CachedUniqueIndexes() [][]string {
    return [][]string{{"Code"}}
}

type UserEntity struct {
    ID        uint64 `orm:"redisCache"`
    Name      string `orm:"required;length=100"`
    Email     string `orm:"length=255"`
    Age       *uint8
    Status    string `orm:"enum=active,blocked;required;enumName=UserStatus"`
    Category  orm.Reference[CategoryEntity] `orm:"required"`
    CreatedAt time.Time
    UpdatedAt time.Time
}

Anatomy of a generated file

The excerpt below is what entities/UserEntity.go looks like (the hex values are computed from the table and column names and differ for every entity; method bodies are omitted).

// Code generated by orm; DO NOT EDIT.

package entities

type userEntityFields struct {
    ID        orm.UintField
    Category  orm.ReferenceField
    CreatedAt orm.TimeField
    UpdatedAt orm.TimeField
    Name      orm.StringField
    Email     orm.NullableStringField
    Age       orm.NullableUintField
    Status    orm.EnumField
}

type userEntityProvider struct {
    tableName        string
    dbCode           string
    redisCode        string
    cacheIndex       string
    redisCachePrefix string
    redisCacheStamp  string
    redisCacheTTL    int
    Fields           userEntityFields
}

var UserEntityProvider = userEntityProvider{
    tableName:        "UserEntity",
    dbCode:           "default",
    redisCode:        "default",
    cacheIndex:       "model.UserEntity",
    redisCachePrefix: "e11dd246:",
    redisCacheStamp:  "e69f360206e0533c",
    redisCacheTTL:    0,
    Fields: userEntityFields{
        ID:       orm.UintField{Column: "ID"},
        Category: orm.ReferenceField{Column: "Category"},
        // ...
    },
}

func (p userEntityProvider) TableName() string { return p.tableName }
func (p userEntityProvider) DBCode() string    { return p.dbCode }

func (p userEntityProvider) RedisCode() string        { return p.redisCode }
func (p userEntityProvider) RedisCachePrefix() string { return p.redisCachePrefix }
func (p userEntityProvider) ClearRedisCache(ctx orm.Context) (int, error)

type userEntitySQLRow struct {
    F0 uint64
    F1 uint64
    F2 time.Time
    F3 time.Time
    F4 string
    F5 sql.NullString
    F6 sql.NullInt64
    F7 string
}

func (p userEntityProvider) OnAfterInsert(engine orm.Engine, handler func(ctx orm.Context, entity *UserEntity) error)
func (p userEntityProvider) OnAfterUpdate(engine orm.Engine, handler func(ctx orm.Context, entity *UserEntity, changes map[string]any) error)
func (p userEntityProvider) OnAfterDelete(engine orm.Engine, handler func(ctx orm.Context, entity *UserEntity) error)

func RegisterUserEntityBeforeInsert(cb func(entity *UserEntity))
func RegisterUserEntityBeforeUpdate(cb func(entity *UserEntity))
func RegisterUserEntityBeforeDelete(cb func(entity *UserEntity))

func (p userEntityProvider) GetByID(ctx orm.Context, id uint64) (entity *UserEntity, err error)
func (p userEntityProvider) GetByIDs(ctx orm.Context, id ...uint64) ([]*UserEntity, error)
func (p userEntityProvider) New(ctx orm.Context) *UserEntity
func (p userEntityProvider) NewWithID(ctx orm.Context, id uint64) *UserEntity
func (p userEntityProvider) SearchOne(ctx orm.Context, query *orm.DBQuery) (*UserEntity, error)
func (p userEntityProvider) SearchMany(ctx orm.Context, query *orm.DBQuery) ([]*UserEntity, error)
func (p userEntityProvider) SearchManyWithTotal(ctx orm.Context, query *orm.DBQuery) ([]*UserEntity, int, error)
func (p userEntityProvider) Count(ctx orm.Context, query *orm.DBQuery) (int, error)

type UserEntity struct {
    ctx                  orm.Context
    id                   uint64
    new                  bool
    deleted              bool
    removed              bool
    snapshot             bool
    originDatabaseValues *userEntitySQLRow
    databaseBind         map[string]any
    originRedisValues    []string
    flushType            uint8
    flushChanges         map[string]any
}

func (e *UserEntity) GetID() uint64

func (e *UserEntity) GetName() string
func (e *UserEntity) SetName(value string) *UserEntity
func (e *UserEntity) GetEmail() string
func (e *UserEntity) SetEmail(value string) *UserEntity
func (e *UserEntity) GetAge() *uint64
func (e *UserEntity) SetAge(value *uint64) *UserEntity
func (e *UserEntity) GetStatus() enums.UserStatus
func (e *UserEntity) SetStatus(value enums.UserStatus) *UserEntity
func (e *UserEntity) GetCategoryID() uint64
func (e *UserEntity) SetCategory(value uint64) *UserEntity
func (e *UserEntity) GetCategory(ctx orm.Context) (reference *CategoryEntity, err error)
func (e *UserEntity) GetCreatedAt() time.Time
func (e *UserEntity) SetCreatedAt(value time.Time) *UserEntity
func (e *UserEntity) GetUpdatedAt() time.Time
func (e *UserEntity) SetUpdatedAt(value time.Time) *UserEntity

The redisCachePrefix, redisCacheStamp and redisCacheTTL provider fields and the entity's originRedisValues exist only for entities tagged orm:"redisCache"; an entity that only declares CachedUniqueIndexes() gets redisCachePrefix alone. An entity tagged orm:"redisCache;cached" also gets a redisAllKey field and the GetAll method. Entities tagged cdc or outbox get the change-feed block. Entities without redisCache get two unexported row readers (entityFromRow, readRows) used by their Search* methods.

Methods whose name starts with Private (PrivateFlush, PrivateFlushed, PrivateSnapshot, PrivateRollback, PrivateAdvance, PrivateIsSnapshot, PrivateIsDeleted, PrivateDatabasePool, PrivateReload, PrivateDelete, PrivateForceDelete (the latter only on FakeDelete entities), PrivateFlushEvent, PrivateGetDatabaseBind, PrivateCacheIndex, PrivateIsNew, PrivateContext, and on cdc/outbox entities PrivateCDCRow, PrivateCDCChange, PrivateColumnValue) implement the orm.Entity interface or internal capabilities used by ctx.Save, ctx.Delete, ctx.Reload and transactions. They are exported only because the ORM lives in another package; never call them yourself.

The Provider

Field descriptors

Provider.Fields holds one typed descriptor per column. Descriptors build the conditions passed to orm.NewQuery().Filter(...) and the sort arguments of SortByASC/SortByDESC:

users, err := entities.UserEntityProvider.SearchMany(ctx,
    orm.NewQuery().
        Filter(
            entities.UserEntityProvider.Fields.Category.Eq(categoryID),
            entities.UserEntityProvider.Fields.Status.Is(enums.UserStatusList.Active),
        ).
        SortByASC(entities.UserEntityProvider.Fields.Name).
        Pager(orm.NewPager(1, 20)),
)
Go fieldDescriptor
uint* (including ID)orm.UintField
*uint*orm.NullableUintField
int* / *int*orm.IntField / orm.NullableIntField
float* / *float*orm.FloatField / orm.NullableFloatField
bool / *boolorm.BoolField / orm.NullableBoolField
time.Time / *time.Timeorm.TimeField / orm.NullableTimeField
string with requiredorm.StringField
string without requiredorm.NullableStringField
string with enum= and requiredorm.EnumField
string with enum= without requiredorm.NullableEnumField
orm.Reference[T] with requiredorm.ReferenceField
orm.Reference[T] without requiredorm.NullableUintField

FakeDelete, []uint8, set= fields, orm.References[T] and JSON struct fields have no descriptor: they cannot be used in Filter. Fields of embedded or nested structs are flattened with the nested field name as prefix (TestSubSize for TestSub.Size), anonymous embedding adds no prefix. The available condition methods are documented in Search.

Metadata methods and interfaces

Every provider implements orm.EntityProvider:

type EntityProvider interface {
    TableName() string
    DBCode() string
}

Entities with orm:"redisCache" or with CachedUniqueIndexes() additionally implement orm.RedisCacheEntityProvider (an entity that only caches unique index lookups still needs a way to clear those keys):

type RedisCacheEntityProvider interface {
    RedisCode() string
    RedisCachePrefix() string
    ClearRedisCache(ctx Context) (int, error)
}

providers.go collects every provider so that tooling can iterate over them and pick the capabilities it needs:

for _, provider := range entities.AllProviders {
    if cached, ok := provider.(orm.RedisCacheEntityProvider); ok {
        deleted, err := cached.ClearRedisCache(ctx)
        // ...
    }
}

Loading, factory and search methods

MethodNotes
GetByID(ctx, id) (*E, error)Context cache, then Redis row cache, then MySQL. A missing row is a *orm.NotFoundError. See CRUD.
GetByIDs(ctx, id ...uint64) ([]*E, error)Input order preserved, duplicates collapsed, missing ids skipped.
New(ctx) *ENew entity with a snowflake ID from ctx.Engine().NextID(); never fails.
NewWithID(ctx, id) *ENew entity with a caller-chosen ID.
SearchOne(ctx, *DBQuery) (*E, error)Uses the cached unique index fast path when the filter matches one exactly. No match is a *orm.NotFoundError.
SearchMany(ctx, *DBQuery) ([]*E, error)
SearchManyWithTotal(ctx, *DBQuery) ([]*E, int, error)
Count(ctx, *DBQuery) (int, error)
GetAll(ctx) ([]*E, error)Only with orm:"redisCache;cached": every row, with the set of ids cached in Redis.
NewSnapshot(id, fill func(*E)) *EOnly with cdc or outbox: a read-only entity built in memory, for tests that hand-build events. See Testing.

A miss is detected with errors.AsType:

user, err := entities.UserEntityProvider.GetByID(ctx, id)
if _, missing := errors.AsType[*orm.NotFoundError](err); missing {
    // no such row
} else if err != nil {
    return err
}

orm.NotFoundError has the fields Entity (the table name) and ID (0 for SearchOne and for an unset reference). There is no found flag and no MustGet* variant.

GetByID, GetByIDs, New, NewWithID are described in CRUD; the Search* and Count methods in Search. There is no generated Delete on the provider and no GetBy<Index> methods: deleting goes through ctx.Delete, unique index lookups through SearchOne.

Lifecycle registration

OnAfterInsert, OnAfterUpdate and OnAfterDelete register one post-commit handler per event on the Engine. The package-level Register<Entity>BeforeInsert, Register<Entity>BeforeUpdate and Register<Entity>BeforeDelete functions register synchronous callbacks that run before the SQL statement is built. Both families are documented in Lifecycle Callbacks.

The Entity

An entity remembers the Context that created or loaded it, its ID, its latest loaded or successfully saved values (originDatabaseValues / originRedisValues) and the columns changed since (databaseBind). A successful Save advances that baseline immediately after SQL execution, even before a transaction commits. The ORM keeps separate snapshots for deferred events and rollback; see Transactions. The entity can only be saved on its own context; see CRUD for the write rules.

Getters and setters

Every setter returns the entity, so calls chain:

user := entities.UserEntityProvider.New(ctx)
user.SetName("Alice").
    SetEmail("alice@example.com").
    SetStatus(enums.UserStatusList.Active).
    SetCategory(category.GetID())
Go fieldGetterSetter
uint*Get<F>() uint64Set<F>(value uint64)
int*Get<F>() int64Set<F>(value int64)
float*Get<F>() float64Set<F>(value float64)
boolGet<F>() boolSet<F>(value bool)
time.TimeGet<F>() time.TimeSet<F>(value time.Time) — truncated to the second (orm:"time") or to the day (date)
string with requiredGet<F>() stringSet<F>(value string)
string without requiredGet<F>() string — "" for NULLSet<F>(value string) — "" stores NULL
*uint*Get<F>() *uint64Set<F>(value *uint64) — nil stores NULL
*int*Get<F>() *int64Set<F>(value *int64)
*float*Get<F>() *float64Set<F>(value *float64)
*boolGet<F>() *boolSet<F>(value *bool)
*time.TimeGet<F>() *time.TimeSet<F>(value *time.Time)
[]uint8Get<F>() []uint8 — nil for NULLSet<F>(value []uint8)
enum with requiredGet<F>() enums.XSet<F>(value enums.X)
enum without requiredGet<F>() *enums.XSet<F>(value *enums.X)
set with requiredGet<F>() []enums.XSet<F>(value ...enums.X) — stored sorted, comma-joined
set without requiredGet<F>() []enums.X — nil for NULLSet<F>(value ...enums.X) — no arguments stores NULL
JSON struct *pkg.TGet<F>() *pkg.TSet<F>(value *pkg.T) — nil stores NULL
orm.Reference[T]Get<F>ID() uint64 (0 for NULL), Get<F>(ctx) (*T, error)Set<F>(value uint64) — 0 stores NULL on an optional reference
orm.References[T]Get<F>IDs() []uint64, Get<F>(ctx) ([]*T, error)Set<F>IDs(ids []uint64) — empty stores []; nil stores NULL on an optional field

Narrow Go integers widen in the API: a uint32 column has Get<F>() uint64, a *uint8 column has Get<F>() *uint64. The ID field has only GetID().

Reference getters load lazily through the referenced provider: Get<F>(ctx) returns a *orm.NotFoundError (with ID 0) when the ID is 0, otherwise the result of <Ref>Provider.GetByID(ctx, id), which is a *orm.NotFoundError carrying the ID when the referenced row is missing. Get<F>(ctx) on a References[T] field calls <Ref>Provider.GetByIDs(ctx, ids...).

Change events

Entities tagged orm:"cdc" or orm:"outbox" get a change-feed block at the end of their file. With orm:"cdc" on UserEntity:

// UserEntityDirtyEvent is one UserEntity row change as a listener receives it.
// Before and After are read-only entities: Insert sets only After, Delete only Before.
type UserEntityDirtyEvent = orm.DirtyEvent[UserEntity]

// One entry per column, in SQL row order, with the column's wire encoding.
var userEntityCDCColumns = []orm.CDCColumn{
    {Name: "ID", Kind: orm.CDCNumber},
    {Name: "Category", Kind: orm.CDCNumber},
    {Name: "CreatedAt", Kind: orm.CDCDate},
    // ...
}

func (r *userEntitySQLRow) cdcFields() []any

func (e *UserEntity) PrivateCDCRow(withChanges bool) map[string]any
func (e *UserEntity) PrivateCDCChange(op orm.DirtyOp) orm.CDCChange
func (e *UserEntity) PrivateColumnValue(column string) any

func decodeUserEntityCDCRow(ctx orm.Context, image map[string]json.RawMessage) (*UserEntity, error)

func (p userEntityProvider) NewSnapshot(id uint64, fill func(e *UserEntity)) *UserEntity

func init() {
    orm.RegisterCDCEntity[UserEntity]("UserEntity")
}
MemberPurpose
<Entity>DirtyEventThe event type handlers receive; Before/After are *<Entity>.
<entity>CDCColumns, cdcFields()The codec between the generated SQL row and the Debezium row image (orm.EncodeCDCRow / orm.DecodeCDCRow). Kind is one of orm.CDCNumber, CDCBool, CDCFloat, CDCDecimal (with Scale), CDCDatetime, CDCDate, CDCString, CDCBytes.
PrivateCDCRow, PrivateCDCChangeBuild the row image and the change (c/u/d) one write puts on the feed.
PrivateColumnValueLets orm.WatchFields compare one column of two images.
decode<Entity>CDCRowTurns a row image into a read-only entity; the generated On<Entity> consumer methods pass it to orm.BuildDispatch.
NewSnapshotBuilds a read-only entity in memory through the setters, for tests.
init()Puts the entity on the feed under its table name.

See Entity Events for the envelope and the event semantics.

Elasticsearch fields

For every document registered with registry.RegisterElasticsearchDocument(...), Generate also writes <outputDirectory>/esfields/<document_name>.go (package esfields) with the document's typed field names and its mappings.properties map. The directory is wiped and rewritten on every run, like the output directory itself. See Elasticsearch.

Enums

Every enum= or set= field produces a file in the enums package. The type name is the enumName= tag value; without it the generator derives one from the entity and field name. Several fields — in the same or in different entities — may share one enum by giving them the same enumName; a field may even reference an enum that is defined elsewhere with orm:"enumName=UserStatus" alone.

// Code generated by orm; DO NOT EDIT.

package enums

type UserStatus string

var UserStatusList = struct {
    Active  UserStatus
    Blocked UserStatus
}{
    Active:  "active",
    Blocked: "blocked",
}

func (e UserStatus) Valid() bool {
    switch e {
    case "active", "blocked":
        return true
    }
    return false
}

func (e UserStatus) Values() []UserStatus {
    return []UserStatus{"active", "blocked"}
}

The constant names are the values with the first letter of every _-separated segment capitalised (in_review becomes InReview). A required enum or set field of a freshly created entity defaults to the first declared value, so entities.UserEntityProvider.New(ctx).GetStatus() is enums.UserStatusList.Active.

Edit this page
Last Updated: 11/16/26, 1:49 PM
Prev
MySQL Indexes
Next
Engine