loadconf

English | 繁體中文

Declare a struct and write a config file; get back validated settings, a logger, and each driver's own client (*sql.DB, *gorm.DB, redis.UniversalClient, *kgo.Client, …).

Every connector is a separate Go module, so a service that only uses Redis does not depend on the Kafka or Oracle drivers. Requires Go 1.26 or later.

Contents

• Install
• Examples
  1. One config file
  2. One file per environment
  3. Environment variables and secrets
  4. Several databases
  5. Configuration in etcd
  6. Driver settings
  7. Read replicas
  8. Logging to several outputs
  9. Hot reload and reconnecting
  10. Configuration for each layer
  11. A file path of your own, and struct tags you already have
  12. A format of your own
  13. Types and checks of your own
  14. A source of your own
• Reference
  • Sources and precedence
  • Which sources read ENV
  • Environment variable names
  • Struct tags
  • Open and New
  • Connectors
  • Logger
• Versioning

Install

# All modules are released together under one version: request the same version for each.
go get github.com/linzeyan/loadconf/config@v0.1.0 \
  github.com/linzeyan/loadconf/logger@v0.1.0 \
  github.com/linzeyan/loadconf/connector/conn_gorm@v0.1.0

The import path is the module path; the package name is its last element.

Import path	Provides
github.com/linzeyan/loadconf/config	Loading, and the settings types of every connection (config.MySQL, config.Redis, …)
github.com/linzeyan/loadconf/config/etcd	etcd sources; a separate module, so services without etcd do not depend on its client
github.com/linzeyan/loadconf/logger	An slog logger and its outputs
github.com/linzeyan/loadconf/logger/logger_zap, …/logger_zerolog, …/sink_otlp	The same configuration as a zap or zerolog logger; the OTLP output
github.com/linzeyan/loadconf/connector/conn_<driver>	One module per driver; see Connectors

config/example.yaml lists every built-in setting with its default, plus three custom sections, commented in Chinese. config/schema.json describes the built-in settings for editor completion and checks; with the YAML extension of VS Code, add # yaml-language-server: $schema=https://raw.githubusercontent.com/linzeyan/loadconf/main/config/schema.json to the top of a config file. Both are generated from the Go types by make generate.

Examples

Each example opens with Before: the same task as a popular open-source project writes it without loadconf, its config file and its code condensed from the linked files, and a table of what differs. With loadconf follows, with the config file and the code; it adds one thing to an earlier example and names the change.

1 One config file

Before, from gin-vue-admin (config.yaml, config/db_list.go, config/gorm_mysql.go, core/viper.go, initialize/gorm_mysql.go):

# config.yaml
mysql:
    path: ""
    port: ""
    config: ""                # the DSN's query string
    db-name: ""
    username: ""
    password: ""
    max-idle-conns: 10
    max-open-conns: 100
    conn-max-lifetime: 0      # seconds
    log-mode: ""
    log-zap: false

// config/db_list.go
type GeneralDB struct {
	Path            string `mapstructure:"path" json:"path" yaml:"path"`
	Port            string `mapstructure:"port" json:"port" yaml:"port"`
	Config          string `mapstructure:"config" json:"config" yaml:"config"` // the DSN's query string
	Dbname          string `mapstructure:"db-name" json:"db-name" yaml:"db-name"`
	Username        string `mapstructure:"username" json:"username" yaml:"username"`
	Password        string `mapstructure:"password" json:"password" yaml:"password"`
	MaxIdleConns    int    `mapstructure:"max-idle-conns" json:"max-idle-conns" yaml:"max-idle-conns"`
	MaxOpenConns    int    `mapstructure:"max-open-conns" json:"max-open-conns" yaml:"max-open-conns"`
	ConnMaxLifetime int    `mapstructure:"conn-max-lifetime" json:"conn-max-lifetime" yaml:"conn-max-lifetime"` // seconds
	// …
}

// config/gorm_mysql.go
func (m *Mysql) Dsn() string {
	return m.Username + ":" + m.Password + "@tcp(" + m.Path + ":" + m.Port + ")/" + m.Dbname + "?" + m.Config
}

// core/viper.go
v := viper.New()
v.SetConfigFile(config)
v.SetConfigType("yaml")
if err := v.ReadInConfig(); err != nil {
	panic(fmt.Errorf("fatal error config file: %w", err))
}
if err := v.Unmarshal(&global.GVA_CONFIG); err != nil {
	panic(fmt.Errorf("fatal error unmarshal config: %w", err))
}

// initialize/gorm_mysql.go
db, err := gorm.Open(mysql.New(mysql.Config{DSN: m.Dsn()}), internal.Gorm.Config(m.GeneralDB))
if err != nil {
	panic(err)
}
sqlDB, _ := db.DB()
sqlDB.SetMaxIdleConns(m.MaxIdleConns)
sqlDB.SetMaxOpenConns(m.MaxOpenConns)
sqlDB.SetConnMaxLifetime(time.Duration(m.ConnMaxLifetime) * time.Second)

	gin-vue-admin	loadconf
Settings type	written per project, with three tags on every field	config.MySQL; your own fields need no tags
DSN	concatenated in Dsn(); extra parameters are one raw query string	built from the fields; parameters are a map under params
Durations	an int of seconds, multiplied in code	5m, 90s
Pool	set by hand; a key left out is 0, and SetMaxIdleConns(0) keeps no idle connections	applied from the file; a key left out keeps database/sql's default
Mistakes	a misspelled key is ignored; an empty host is found when connecting	a misspelled key logs a warning; every invalid value is reported at once, with its key path
Connecting	no deadline unless the query string sets timeout	a ping within ping_timeout (3s)

With loadconf. config.yaml:

# A key is the snake_case name of its field, with product names kept whole:
# MySQL -> mysql, MaxOpenConns -> max_open_conns. Matching ignores case.
port: 8080
mysql:
  host: 127.0.0.1         # port: 3306 unless set
  user: app
  password: dev-only
  database: shop
  max_open_conns: 20      # unset or 0 keeps database/sql's default: no limit
  conn_max_lifetime: 5m

main.go:

package main

import (
	"context"
	"log"

	"github.com/linzeyan/loadconf/config"
	"github.com/linzeyan/loadconf/connector/conn_gorm"
)

// Fields need no tags.
type Config struct {
	Port  int `default:"8080"` // used when no source sets port
	MySQL config.MySQL
}

func main() {
	ctx := context.Background()
	// The extension selects the format: .yaml, .yml, .json or .toml.
	// Load merges the sources, applies defaults, decodes, then validates. It
	// reports every problem at once, each with its key path:
	//   config: decode: mysql.max_open_conns: invalid integer "many"
	//   mysql.port: invalid integer "abc"
	// A key that matches no field, usually a typo, only logs a warning:
	//   WARN config keys match no field and are ignored source=file(config.yaml) keys=[mysql.hots]
	// With config.WithStrict() it is an error instead.
	cfg, err := config.Load[Config](ctx, config.From(config.File("config.yaml")))
	if err != nil {
		log.Fatal(err)
	}

	// OpenMySQL builds the pool and pings it within ping_timeout (default
	// 3s). If the ping fails, it closes the pool and returns the error.
	db, err := conn_gorm.OpenMySQL(ctx, cfg.MySQL)
	if err != nil {
		log.Fatal(err)
	}
	defer conn_gorm.Close(db)

	log.Printf("listening on :%d", cfg.Port)
}

2 One file per environment

Before, from gin-vue-admin (config.yaml, config.docker.yaml, Dockerfile, core/viper.go, core/internal/constant.go). The repository ships two complete files; the Dockerfile runs ./server -c config.docker.yaml.

# config.yaml: 291 lines
redis:
    useCluster: false
    addr: 127.0.0.1:6379
    username: ""
    password: ""
    db: 0
# …
system:
    # …
    disable-auto-migrate: false
mysql:
    # …
    conn-max-lifetime: 0

# config.docker.yaml: 282 lines, a copy of config.yaml to change addresses
redis:
    useCluster: false
    addr: 177.7.0.14:6379
    # username is missing
    password: ""
    db: 0
# … identical but for addresses
system:
    # …
    # disable-auto-migrate is missing
mysql:
    # …
    # conn-max-lifetime is missing, here and in the other five database sections

// getConfigPath picks one file, and only that file is read.
func getConfigPath() (config string) {
	flag.StringVar(&config, "c", "", "choose config file.")
	flag.Parse()
	if config != "" {
		return
	}
	if env := os.Getenv("GVA_CONFIG"); env != "" {
		return env
	}
	switch gin.Mode() {
	case gin.DebugMode:
		config = "config.debug.yaml"
	case gin.ReleaseMode:
		config = "config.release.yaml"
	case gin.TestMode:
		config = "config.test.yaml"
	}
	if _, err := os.Stat(config); err != nil {
		config = "config.yaml"
	}
	return
}

	gin-vue-admin	loadconf
Files read	one; each environment's file repeats the shared keys	app.yaml, then app.$ENV.yaml over it, key by key
A key added later	added to each copy by hand; config.docker.yaml already lacks three	added once to app.yaml
An environment without its own file	runs on config.yaml	Load fails, naming the file it looked for
Chosen by	-c, then $GVA_CONFIG, then gin's mode	$ENV; $CONFIG_PATH names a single file

With loadconf, changes from example 1: config.File("config.yaml") becomes config.Profile("app").

configs/
├── app.yaml        # shared by every environment; may be absent
├── app.dev.yaml    # ENV=dev, and ENV unset
└── app.prod.yaml   # ENV=prod

# configs/app.yaml
port: 8080
mysql:
  user: app
  database: shop
  max_open_conns: 20

# configs/app.dev.yaml: merged into app.yaml key by key
mysql:
  host: 127.0.0.1
  password: dev-only

# configs/app.prod.yaml: the password comes from the environment (example 3)
mysql:
  host: db.prod.internal
  max_open_conns: 100     # overrides the 20 of app.yaml; user and database are kept

package main

import (
	"context"
	"log"

	"github.com/linzeyan/loadconf/config"
	"github.com/linzeyan/loadconf/connector/conn_gorm"
)

type Config struct {
	Port  int `default:"8080"`
	MySQL config.MySQL
}

func main() {
	ctx := context.Background()
	// Reads configs/app.yaml, then configs/app.$ENV.yaml over it. Each load
	// logs the files it read:
	//   INFO config loaded sources="[file(configs/app.yaml) file(configs/app.dev.yaml)]"
	cfg, err := config.Load[Config](ctx, config.From(config.Profile("app")))
	if err != nil {
		log.Fatal(err)
	}

	db, err := conn_gorm.OpenMySQL(ctx, cfg.MySQL)
	if err != nil {
		log.Fatal(err)
	}
	defer conn_gorm.Close(db)

	log.Printf("listening on :%d", cfg.Port)
}

Run with	Files read	mysql.host	mysql.max_open_conns
ENV unset, or ENV=dev	app.yaml, then app.dev.yaml	127.0.0.1	20
ENV=prod	app.yaml, then app.prod.yaml	db.prod.internal	100
ENV=stage	none; Load fails: no config file app.stage{.yaml,.yml,.json,.toml} in ./configs
CONFIG_PATH=/etc/app/app.yaml	that file only; ENV is ignored	from that file	from that file

Profile options:

config.Profile("app", config.ProfileDir("/etc/app"))              // directory of the files; default ./configs
config.Profile("app", config.ProfileEnvs("dev", "stage", "prod")) // any other ENV fails: unsupported ENV="prdo" (want one of dev, stage, prod)
config.Profile("app", config.ProfileEnvVar("APP_ENV"))            // read APP_ENV instead of ENV
config.Profile("app", config.ProfileDefaultEnv("local"))          // environment when the variable is unset; default dev
config.Profile("app", config.ProfilePathEnvVar("APP_CONFIG"))     // variable naming a single file; default CONFIG_PATH

3 Environment variables and secrets

Before, from go-clean-arch (example.env, app/main.go). There is no config file; every setting is a variable, and example.env is copied to .env:

# example.env
DEBUG = True
SERVER_ADDRESS = ":9090"
CONTEXT_TIMEOUT = 2
DATABASE_HOST = "localhost"
DATABASE_PORT = "3306"
DATABASE_USER = "user"
DATABASE_PASS = "password"
DATABASE_NAME = "article"

func init() {
	err := godotenv.Load()
	if err != nil {
		log.Fatal("Error loading .env file")
	}
}

func main() {
	dbHost := os.Getenv("DATABASE_HOST")
	dbPort := os.Getenv("DATABASE_PORT")
	dbUser := os.Getenv("DATABASE_USER")
	dbPass := os.Getenv("DATABASE_PASS")
	dbName := os.Getenv("DATABASE_NAME")
	connection := fmt.Sprintf("%s:%s@tcp(%s:%s)/%s", dbUser, dbPass, dbHost, dbPort, dbName)
	// …
	timeoutStr := os.Getenv("CONTEXT_TIMEOUT")
	timeout, err := strconv.Atoi(timeoutStr)
	if err != nil {
		log.Println("failed to parse timeout, using default timeout")
		timeout = defaultTimeout
	}
	timeoutContext := time.Duration(timeout) * time.Second
	// …
}

	go-clean-arch	loadconf
.env	required: without the file the process exits, in production too	DotEnvOptional(): read when present
Reading	one os.Getenv per setting	config.Env("APP") maps every APP_ variable to its key
Config files	none: every setting is a variable	files and variables together; a variable overrides one key
Types	parsed by hand; CONTEXT_TIMEOUT=2s logs a line and runs with the default	decoded by the field's type; a bad value fails Load, naming the key
An unset variable	an empty string, passed into the DSN	a required field fails Load
Password	a plain string	config.Secret: ****** under every fmt verb

With loadconf, changes from example 2: config.DotEnv and config.Env join the sources, and PaymentKey is a config.Secret. The files of example 2 keep the settings shared by every deployment; variables carry only what differs, such as passwords.

# configs/app.yaml: unchanged from example 2, and free of secrets
port: 8080
mysql:
  user: app
  database: shop
  max_open_conns: 20

package main

import (
	"context"
	"log"

	"github.com/linzeyan/loadconf/config"
	"github.com/linzeyan/loadconf/connector/conn_gorm"
)

type Config struct {
	// env:"PORT" reads $PORT after all sources, so the port a platform sets
	// wins over the files and over APP_PORT.
	Port int `default:"8080" env:"PORT"`
	// The password fields of config.MySQL are config.Secret already.
	MySQL config.MySQL
	// A non-empty Secret prints as ****** under every fmt verb; Value()
	// returns the real string.
	PaymentKey  config.Secret
	PaymentHost string `default:"https://pay.example.com"`
}

func main() {
	ctx := context.Background()
	// A later source overrides an earlier one, key by key.
	cfg, err := config.Load[Config](ctx, config.From(
		config.Profile("app"),
		// .env in the working directory, for local development. Without
		// DotEnvOptional, a missing file is an error. Names map to keys as
		// in Env below.
		config.DotEnv(".env", "APP", config.DotEnvOptional()),
		// APP_MYSQL__PASSWORD sets mysql.password: "__" separates levels, a
		// single "_" stays in the key. An empty value counts as unset.
		config.Env("APP"),
	))
	if err != nil {
		log.Fatal(err)
	}
	// Prints ... Password:****** ... PaymentKey:******
	log.Printf("config: %+v", *cfg)

	db, err := conn_gorm.OpenMySQL(ctx, cfg.MySQL)
	if err != nil {
		log.Fatal(err)
	}
	defer conn_gorm.Close(db)

	pay(cfg.PaymentHost, cfg.PaymentKey.Value())
}

func pay(host, key string) {}

# .env: local development only; keep it out of version control
APP_MYSQL__PASSWORD=dev-only
APP_PAYMENT_KEY=test-key

# Production environment
ENV=prod                      # selects configs/app.prod.yaml
APP_MYSQL__PASSWORD=s3cret    # mysql.password
APP_PAYMENT_KEY=live-key      # payment_key
PORT=9090                     # the env tag of Port; APP_PORT=9090 also works, but PORT wins if both are set

4 Several databases

Before, from gin-vue-admin (config.yaml, initialize/db_list.go, global/global.go):

# The main database is under mysql; the others go in a list beside it.
db-list:
  - disable: true
    type: ""            # mysql, pgsql, mssql or oracle
    alias-name: ""      # must be unique in db-list
    path: ""
    port: ""
    db-name: ""
    username: ""
    password: ""
    max-idle-conns: 10
    max-open-conns: 100

// initialize/db_list.go
func DBList() {
	dbMap := make(map[string]*gorm.DB)
	for _, info := range global.GVA_CONFIG.DBList {
		if info.Disable {
			continue
		}
		switch info.Type {
		case "mysql":
			dbMap[info.AliasName] = GormMysqlByConfig(config.Mysql{GeneralDB: info.GeneralDB})
		case "pgsql":
			dbMap[info.AliasName] = GormPgSqlByConfig(config.Pgsql{GeneralDB: info.GeneralDB})
		// mssql, oracle …
		default:
			continue
		}
	}
	global.GVA_DBList = dbMap
}

// global/global.go
func MustGetGlobalDBByDBName(dbname string) *gorm.DB {
	lock.RLock()
	defer lock.RUnlock()
	db, ok := GVA_DBList[dbname]
	if !ok || db == nil {
		panic("db no init")
	}
	return db
}

	gin-vue-admin	loadconf
Declaring	mysql for the main database, db-list for the others; Redis likewise has redis and redis-list	config.Named[config.MySQL]: mysql holds every instance by name
Kinds	one list for every kind; type picks the constructor, and an unknown type is skipped without a word	one typed field per kind
Names	unique by a comment; a repeated alias-name replaces the earlier database	a repeated name fails Load
Changing one instance at deploy time	edit the file	APP_MYSQL__REPORT__PASSWORD=…
Getting one	a global map	MustGet, Get and All on the loaded config

With loadconf, changes from example 3: config.MySQL becomes config.Named[config.MySQL], and the code picks instances by name.

# configs/app.yaml
mysql:                   # a map: each key is an instance name
  orders:
    host: db1
    user: app
    database: orders
  report:
    host: db2
    user: report
    database: report
redis:                   # or a list whose items carry a name
  - name: cache
    addrs: ["r1:6379"]
  - name: session
    addrs: ["r2:6379", "r3:6379"]   # more than one address: a cluster client
# Inside a flow mapping {...}, quote host:port: {addrs: [r1:6379]} parses
# r1:6379 as a map. Block style, as above, needs no quotes.

package main

import (
	"context"
	"log"

	"github.com/redis/go-redis/v9"

	"github.com/linzeyan/loadconf/config"
	"github.com/linzeyan/loadconf/connector/conn_gorm"
	"github.com/linzeyan/loadconf/connector/conn_redis"
)

type Config struct {
	// Named holds instances by name. With exactly one database,
	// config.MySQL is enough (examples 1 to 3).
	MySQL config.Named[config.MySQL]
	Redis config.Named[config.Redis]
}

func main() {
	ctx := context.Background()
	cfg, err := config.Load[Config](ctx, config.From(config.Profile("app"), config.Env("APP")))
	if err != nil {
		log.Fatal(err)
	}

	// MustGet panics when the name is missing, which suits start-up.
	// Names match case-insensitively.
	orders, err := conn_gorm.OpenMySQL(ctx, cfg.MySQL.MustGet("orders"))
	if err != nil {
		log.Fatal(err)
	}
	defer conn_gorm.Close(orders)

	// Get reports whether the instance exists, for optional ones.
	if reportCfg, ok := cfg.MySQL.Get("report"); ok {
		report, err := conn_gorm.OpenMySQL(ctx, reportCfg)
		if err != nil {
			log.Fatal(err)
		}
		defer conn_gorm.Close(report)
	}

	// All yields every instance, sorted by name.
	caches := map[string]redis.UniversalClient{}
	for name, rc := range cfg.Redis.All() {
		c, err := conn_redis.Open(ctx, rc)
		if err != nil {
			log.Fatalf("redis %s: %v", name, err)
		}
		defer c.Close()
		caches[name] = c
	}
}

Sources merge instance by instance, so an environment variable can change one field of one instance, or add an instance:

APP_MYSQL__REPORT__PASSWORD=s3cret   # report keeps its other fields
APP_MYSQL__ARCHIVE__HOST=10.0.0.3    # a new instance "archive", defined by the environment alone

5 Configuration in etcd

Before, from viper's README. The configuration is one YAML document stored as the value of the etcd key /config/hugo.yml; the README does not show the document.

import _ "github.com/spf13/viper/remote"

var runtime_viper = viper.New()

runtime_viper.AddRemoteProvider("etcd", "http://127.0.0.1:4001", "/config/hugo.yml")
runtime_viper.SetConfigType("yaml") // because there is no file extension in a stream of bytes

err := runtime_viper.ReadRemoteConfig()
runtime_viper.Unmarshal(&runtime_conf)

go func() {
	for {
		time.Sleep(time.Second * 5) // delay after each request

		err := runtime_viper.WatchRemoteConfig()
		if err != nil {
			log.Errorf("unable to read remote config: %v", err)
			continue
		}

		runtime_viper.Unmarshal(&runtime_conf)
	}
}()

	viper	loadconf
etcd client	built by viper from an endpoint string, with no parameter for a username or password	the clientv3.Client you configure
Layout	one key holding the whole document	one document (etcd.Key) or one key per field (etcd.Prefix)
Layers	one remote document	several keys and prefixes merged in order, then config.Env
Changes	a loop you write: sleep, read, Unmarshal into the struct in use	Watch, as for files; resumes from the revision of the last read

With loadconf, changes from example 4: the configuration is read from etcd. The program must know where etcd is before it can read from it, so it loads in two stages.

# The environment says where etcd is; nothing else is set here.
APP_ETCD__ENDPOINTS=etcd1:2379,etcd2:2379
APP_ETCD__USERNAME=order-service
APP_ETCD__PASSWORD=s3cret

# The value of the etcd key /order-service/prod/config.yaml: the same document
# as configs/app.yaml of example 4
port: 9090
mysql:
  orders:
    host: db1
    user: app
    database: orders

package main

import (
	"cmp"
	"context"
	"log"
	"os"
	"time"

	clientv3 "go.etcd.io/etcd/client/v3"

	"github.com/linzeyan/loadconf/config"
	"github.com/linzeyan/loadconf/config/etcd"
	"github.com/linzeyan/loadconf/connector/conn_gorm"
)

// Bootstrap is what the process needs to reach etcd.
type Bootstrap struct {
	Etcd EtcdClient
}

type EtcdClient struct {
	Endpoints []string `validate:"min=1"`
	Username  string
	Password  config.Secret
}

type Config struct {
	Port  int `default:"8080"`
	MySQL config.Named[config.MySQL]
}

func main() {
	ctx := context.Background()
	// ENV selects only Profile's file names. etcd keys are read exactly as
	// written, so the environment has to be part of the key. cmp.Or applies
	// Profile's rule (unset means dev), so files and etcd agree.
	env := cmp.Or(os.Getenv("ENV"), "dev")

	// Stage 1, from the environment: APP_ETCD__ENDPOINTS=etcd1:2379,etcd2:2379
	boot, err := config.Load[Bootstrap](ctx, config.From(config.Env("APP")))
	if err != nil {
		log.Fatal(err)
	}
	// With a Username, clientv3.New authenticates at once and fails if etcd
	// is unreachable. Without one, New does no I/O and the first read fails.
	cli, err := clientv3.New(clientv3.Config{
		Endpoints:   boot.Etcd.Endpoints,
		Username:    boot.Etcd.Username,
		Password:    boot.Etcd.Password.Value(),
		DialTimeout: 5 * time.Second,
	})
	if err != nil {
		log.Fatal(err)
	}
	defer cli.Close()

	// Stage 2. Each etcd read gives up after 5s (etcd.Timeout changes it):
	//   config: load etcd-key(/order-service/prod/config.yaml): context deadline exceeded
	// A missing key fails too, unless the source has etcd.Optional():
	//   config: load etcd-key(/order-service/prod/config.yaml): key /order-service/prod/config.yaml not found
	cfg, err := config.Load[Config](ctx, config.From(
		// One key holding the whole document; the .yaml suffix sets the format.
		etcd.Key(cli, "/order-service/"+env+"/config.yaml"),
		config.Env("APP"), // still overrides single fields
	))
	if err != nil {
		log.Fatal(err)
	}

	db, err := conn_gorm.OpenMySQL(ctx, cfg.MySQL.MustGet("orders"))
	if err != nil {
		log.Fatal(err)
	}
	defer conn_gorm.Close(db)
}

Three ways to lay out the keys:

Source	Stored in etcd	Suits
etcd.Key(cli, "/order-service/prod/config.yaml")	the whole YAML document in one key	reviewing and changing the configuration as one document
etcd.Key(cli, "/sport/prod", etcd.Format("json"))	the whole JSON document in one key without a suffix	an existing /{app}/{env} JSON layout; keys such as "MySQL" match too
etcd.Prefix(cli, "/order-service/prod/")	one key per field, as below	fields maintained by different people or tools

# etcd.Prefix: the key after the prefix, split on "/", is the key path.
# Values are strings, converted as environment variables are.
/order-service/prod/port                    9090
/order-service/prod/mysql/orders/host       db1
/order-service/prod/mysql/orders/password   s3cret
/order-service/prod/redis/cache/addrs       r1:6379,r2:6379

// Shared and per-environment layers, like app.yaml and app.$ENV.yaml:
cfg, err := config.Load[Config](ctx, config.From(
	etcd.Prefix(cli, "/order-service/common/"),  // every environment
	etcd.Prefix(cli, "/order-service/"+env+"/"), // this environment; overrides common
	config.Env("APP"),
))

Both source kinds are watchable: under Loader.Watch (example 9), a change to the key or under the prefix triggers a reload. The watch resumes from the revision of the last read, so no change in between is missed.

6 Driver settings

Before, from gin-vue-admin (config.yaml, config/redis.go, initialize/redis.go):

# config.yaml: every key the code reads, and nothing more
redis:
    useCluster: false
    addr: 127.0.0.1:6379      # ignored when useCluster is true
    username: ""
    password: ""
    db: 0
    clusterAddrs:
        - "172.21.0.3:7000"
        - "172.21.0.4:7001"
        - "172.21.0.2:7002"

// config/redis.go
type Redis struct {
	Name         string   `mapstructure:"name" json:"name" yaml:"name"`
	Addr         string   `mapstructure:"addr" json:"addr" yaml:"addr"`
	Username     string   `mapstructure:"username" json:"username" yaml:"username"`
	Password     string   `mapstructure:"password" json:"password" yaml:"password"`
	DB           int      `mapstructure:"db" json:"db" yaml:"db"`
	UseCluster   bool     `mapstructure:"useCluster" json:"useCluster" yaml:"useCluster"`
	ClusterAddrs []string `mapstructure:"clusterAddrs" json:"clusterAddrs" yaml:"clusterAddrs"`
}

// initialize/redis.go
if redisCfg.UseCluster {
	client = redis.NewClusterClient(&redis.ClusterOptions{
		Addrs:    redisCfg.ClusterAddrs,
		Username: redisCfg.Username,
		Password: redisCfg.Password,
	})
} else {
	client = redis.NewClient(&redis.Options{
		Addr:     redisCfg.Addr,
		Username: redisCfg.Username,
		Password: redisCfg.Password,
		DB:       redisCfg.DB,
	})
}

	gin-vue-admin	loadconf
Driver settings	the fields above; a pool size or timeout takes a new field and a new line of code	any field of redis.UniversalOptions under options
A misspelled key	ignored	Open fails: unknown options: pool_sise
Cluster	useCluster and a second address list	more than one address in addrs

With loadconf, changes from example 4: driver settings beyond the typed fields go in options or params in the config file; only functions and hooks go in code.

# configs/app.yaml
redis:
  addrs: ["r1:6379"]
  # options decode into the driver's own struct, here redis.UniversalOptions,
  # with snake_case field names. A misspelled key, a wrong type, or a key that
  # has its own field makes Open fail:
  #   unknown options: pool_sise
  #   options.password: set by its own config key, not in options
  # The keys follow the driver version: if an upgrade renames a field, Open
  # fails the same way instead of silently ignoring the setting.
  options: {pool_size: 50, read_timeout: 2s}

kafka:
  brokers: ["k1:9092"]
  # Read only by conn_kafka_sarama: sarama.Config, nested structs as nested maps.
  # conn_kafka_franz fails when options is set; give it kgo options in code.
  options:
    producer: {compression: zstd, retry: {max: 5}}

postgres:
  main:
    host: pg
    # params are appended to the DSN as given (MySQL, PostgreSQL, SQL Server,
    # Oracle, Trino, MongoDB). Keys from environment variables arrive in lower
    # case, so write case-sensitive ones, such as MySQL's parseTime, in a file.
    params: {sslmode: require, application_name: order-service}

package main

import (
	"context"
	"log"

	"github.com/redis/go-redis/v9"
	"github.com/twmb/franz-go/pkg/kgo"

	"github.com/linzeyan/loadconf/config"
	"github.com/linzeyan/loadconf/connector/conn_kafka_franz"
	"github.com/linzeyan/loadconf/connector/conn_redis"
)

type Config struct {
	Redis    config.Redis
	Kafka    config.Kafka
	Postgres config.Named[config.Postgres] // opened as in example 4
}

func main() {
	ctx := context.Background()
	cfg, err := config.Load[Config](ctx, config.From(config.Profile("app"), config.Env("APP")))
	if err != nil {
		log.Fatal(err)
	}

	// A function cannot be written in a file, so it is passed in code. Code
	// options run after the file's settings and override them.
	rdb, err := conn_redis.Open(ctx, cfg.Redis, func(o *redis.UniversalOptions) {
		o.OnConnect = func(ctx context.Context, cn *redis.Conn) error { return nil }
	})
	if err != nil {
		log.Fatal(err)
	}
	defer rdb.Close()

	// franz-go is configured through functions, so its settings are kgo options.
	kc, err := conn_kafka_franz.Open(ctx, cfg.Kafka, kgo.DefaultProduceTopic("events"))
	if err != nil {
		log.Fatal(err)
	}
	defer kc.Close()
}

go doc github.com/linzeyan/loadconf/config.Redis   # every field of a settings type

7 Read replicas

Before, from gorm's documentation (DBResolver). There is no config file: the DSNs and the pool are written in code, and moving them to a file is left to each project.

db, err := gorm.Open(mysql.Open("db1_dsn"), &gorm.Config{})

db.Use(
	dbresolver.Register(dbresolver.Config{
		Replicas: []gorm.Dialector{mysql.Open("db3_dsn"), mysql.Open("db4_dsn")},
		Policy:   dbresolver.RandomPolicy{},
	}).
		SetConnMaxIdleTime(time.Hour).
		SetConnMaxLifetime(24 * time.Hour).
		SetMaxIdleConns(100).
		SetMaxOpenConns(200),
)

	gorm's documentation	loadconf
DSNs	one string per database, built as in example 1	one config per instance; <<: *orders copies the primary's
Pool	one setting for the whole resolver	each instance's own max_open_conns and the rest
Closing	closing db.DB() closes only the primary's pool	conn_gorm.Close closes every pool

With loadconf, changes from example 4: a second instance is attached as a read replica with conn_gorm.WithMySQLReplicas (WithPostgresReplicas for PostgreSQL).

mysql:
  orders: &orders           # &orders names this mapping
    host: db-primary
    user: app
    database: shop
    max_open_conns: 50
  orders_replica:
    <<: *orders             # copies every key of orders,
    host: db-replica        # then overrides these two
    max_open_conns: 100

# The merge above happens inside the file. Environment variables override
# instances by name, so each instance needs its own password.
APP_MYSQL__ORDERS__PASSWORD=s3cret
APP_MYSQL__ORDERS_REPLICA__PASSWORD=s3cret

package main

import (
	"context"
	"log"

	"gorm.io/gorm"
	"gorm.io/plugin/dbresolver"

	"github.com/linzeyan/loadconf/config"
	"github.com/linzeyan/loadconf/connector/conn_gorm"
)

type Config struct {
	MySQL config.Named[config.MySQL]
}

type Order struct {
	ID     uint
	Status string
}

func main() {
	ctx := context.Background()
	cfg, err := config.Load[Config](ctx, config.From(config.Profile("app"), config.Env("APP")))
	if err != nil {
		log.Fatal(err)
	}

	// Each replica is opened like the primary: its own pool settings,
	// parseTime on, and WithSQLOptions and WithMySQLDialector applied. If
	// any fails, OpenMySQL closes what it opened and returns the error.
	db, err := conn_gorm.OpenMySQL(ctx, cfg.MySQL.MustGet("orders"),
		conn_gorm.WithMySQLReplicas(cfg.MySQL.MustGet("orders_replica")))
	if err != nil {
		log.Fatal(err)
	}
	// Close closes the replica pools too; db.DB() returns only the primary's.
	defer conn_gorm.Close(db)

	order := Order{Status: "new"}
	db.Create(&order)
	db.Find(&[]Order{})
	// A replica may not have the row yet: pin a read that must see a write.
	db.Clauses(dbresolver.Write).First(&order, order.ID)
	db.Transaction(func(tx *gorm.DB) error {
		return tx.Model(&order).Update("status", "paid").Error
	})
}

Operation	Runs on
Create, Save, Update, Delete, Exec	the primary
Find, First, Count, Scan	a replica, chosen at random
Raw("SELECT …")	a replica; the primary if the statement ends with FOR UPDATE
any query inside Transaction, reads included	the primary
a read with Clauses(dbresolver.Write)	the primary

8 Logging to several outputs

Before, from gin-vue-admin (config.yaml, config/zap.go, core/zap.go, core/internal/zap_core.go):

zap:
  level: info
  format: console
  director: log
  encode-level: LowercaseColorLevelEncoder
  log-in-console: true
  retention-day: -1

// config/zap.go
func (c *Zap) Levels() []zapcore.Level {
	levels := make([]zapcore.Level, 0, 7)
	level, err := zapcore.ParseLevel(c.Level)
	if err != nil {
		level = zapcore.DebugLevel
	}
	for ; level <= zapcore.FatalLevel; level++ {
		levels = append(levels, level)
	}
	return levels
}

// core/zap.go: one core per level. Each writes log/<date>/<level>.log, and
// stdout too when log-in-console is set.
levels := global.GVA_CONFIG.Zap.Levels()
cores := make([]zapcore.Core, 0, len(levels))
for i := 0; i < len(levels); i++ {
	cores = append(cores, internal.NewZapCore(levels[i]))
}
logger = zap.New(zapcore.NewTee(cores...))

	gin-vue-admin	loadconf
Outputs	stdout and one file per level per day, fixed in code	named outputs: stdout, stderr, file, syslog, gelf, elasticsearch, otlp
Levels	one global level; a misspelled level means debug	a global level and one per output; a misspelled level fails Load
Old files	a directory per day; retention-day removes old ones	rotation by size, with max_backups, max_age and compress
At run time	built once at start	log.Update changes the levels

With loadconf, changes from example 3: config.Log and logger.New are added. One log call writes the record to every output whose level admits it.

# configs/app.yaml (mysql as in example 2)
log:
  level: info               # the minimum for every output
  service: order-service    # adds service=order-service to every record
  # stack_level: error      # default: records at error and above carry a stack field
  outputs:
    stdout:                 # an output's name is also its type, unless type is set
      format: text          # stdout, stderr and file may use text; other outputs always get JSON
    file:
      path: /var/log/order-service/app.log
      max_size: 100MiB      # rotate beyond this size (default 100MiB)
      max_backups: 7
      compress: true
    errors:                 # a second file output, so its type is set explicitly
      type: file
      path: /var/log/order-service/error.log
      level: error          # this output's minimum; the higher of it and log.level applies
    elasticsearch:
      addresses: ["https://es:9200"]
      index: "logs-{service}-{date:2006.01.02}"
      level: warn

# configs/app.dev.yaml: outputs merge by name, so only the differences are written
log:
  level: debug
  outputs:
    file: {path: logs/app.log}         # the directory is created on open
    errors: {path: logs/error.log}
    elasticsearch: {disabled: true}

package main

import (
	"context"
	"log/slog"
	"os"

	"github.com/linzeyan/loadconf/config"
	"github.com/linzeyan/loadconf/connector/conn_gorm"
	"github.com/linzeyan/loadconf/logger"
)

type Config struct {
	Log   config.Log
	MySQL config.MySQL
}

func main() {
	ctx := context.Background()
	cfg, err := config.Load[Config](ctx, config.From(config.Profile("app"), config.Env("APP")))
	if err != nil {
		slog.Error("load config", "error", err) // no logger yet: slog's default writes to stderr
		os.Exit(1)
	}
	// New opens every output, so a bad path fails here, not at the first
	// record. Once running, a failing output (full disk, Elasticsearch down)
	// does not affect the others and never makes a log call fail; its errors
	// go to logger.WithErrorHandler (default: stderr, at most once per output
	// every 10s).
	// stdout, stderr, file, syslog and gelf write on the calling goroutine,
	// so a slow one slows every log call. elasticsearch and otlp queue the
	// records and send them in batches; when the queue is full they drop
	// records and report how many.
	log, err := logger.New(ctx, cfg.Log)
	if err != nil {
		slog.Error("open logger", "error", err)
		os.Exit(1)
	}
	defer log.Close() // flushes the Elasticsearch queue
	// slog, the log package, and connections opened from here on write
	// through log.
	log.SetDefault()

	db, err := conn_gorm.OpenMySQL(ctx, cfg.MySQL)
	if err != nil {
		log.Error("open mysql", "error", err)
		return
	}
	defer conn_gorm.Close(db)

	log.Debug("cache warmed", "keys", 1200)
	log.Info("started")
	log.Warn("upstream slow", "ms", 900)
	log.Error("payment failed", "order", 42)
}

Where each record goes (verified by running this configuration):

Call	stdout	app.log	error.log	Elasticsearch
log.Debug	dev only	dev only
log.Info	✓	✓
log.Warn	✓	✓		outside dev
log.Error	✓	✓	✓	outside dev

APP_LOG__OUTPUTS__ELASTICSEARCH__DISABLED=true   # turns one output off at deploy time

At run time, log.Update changes the levels, global and per output, at once (example 9). Adding or removing an output, or changing a format, takes a new logger. The fields of syslog, gelf and otlp are under Logger.

9 Hot reload and reconnecting

Before, from gin-vue-admin (config.yaml, core/viper.go, initialize/reload.go). An edit to config.yaml while the server runs, and what the code below does with it:

# config.yaml
zap:
    level: debug              # was info: GVA_CONFIG changes; the logger built at start keeps info
mysql:
    path: 10.0.0.2            # GVA_CONFIG changes; GVA_DB keeps the old connection until Reload is called
    max-open-conns: abc       # Unmarshal fails after the keys before it were already written

// core/viper.go
v.WatchConfig()
v.OnConfigChange(func(e fsnotify.Event) {
	fmt.Println("config file changed:", e.Name)
	if err = v.Unmarshal(&global.GVA_CONFIG); err != nil {
		fmt.Println(err)
	}
})

// initialize/reload.go: connections are rebuilt only when Reload is called.
if global.GVA_DB != nil {
	db, _ := global.GVA_DB.DB()
	if err := db.Close(); err != nil {
		return err
	}
}
global.GVA_DB = Gorm()

	gin-vue-admin	loadconf
Applying a change	Unmarshal writes into global.GVA_CONFIG while requests read it: a data race	a new *Config, swapped in atomically; Current() returns one version or the other
An invalid change	printed; the fields that did decode are already changed	nothing is applied; the current configuration stays and OnError runs
Rebuilding a connection	the old one is closed first; if the new one fails, Gorm() panics	open the new one first; on failure, keep the old one
An editor saving in several writes	a reload per write	one reload, 500ms after the last change

With loadconf, changes from example 8: config.Load becomes config.New, and the loader watches the sources. Log levels follow a change immediately; connections are rebuilt in OnChange. The same kind of edit:

# configs/app.yaml
log:
  level: debug              # was info: OnChange calls log.Update, and debug records appear
mysql:
  host: 10.0.0.2            # OnChange opens a pool to 10.0.0.2, then swaps it in
  max_open_conns: abc       # the whole reload is rejected: nothing above is applied, OnError runs
max_items: 50               # loader.Current().MaxItems returns 50 from the next request on

package main

import (
	"context"
	"log/slog"
	"os"
	"os/signal"
	"reflect"
	"sync/atomic"
	"time"

	"gorm.io/gorm"

	"github.com/linzeyan/loadconf/config"
	"github.com/linzeyan/loadconf/connector/conn_gorm"
	"github.com/linzeyan/loadconf/logger"
)

type Config struct {
	Log      config.Log
	MySQL    config.MySQL
	MaxItems int `default:"100"`
}

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
	defer stop()

	loader := config.New[Config](config.From(config.Profile("app"), config.Env("APP")))
	cfg, err := loader.Load(ctx)
	if err != nil {
		slog.Error("load config", "error", err)
		os.Exit(1)
	}
	log, err := logger.New(ctx, cfg.Log)
	if err != nil {
		slog.Error("open logger", "error", err)
		os.Exit(1)
	}
	defer log.Close()
	log.SetDefault()

	first, err := conn_gorm.OpenMySQL(ctx, cfg.MySQL)
	if err != nil {
		log.Error("open mysql", "error", err)
		return
	}
	// The rest of the program reads the connection through db.Load(), so a
	// swap reaches every later query.
	var db atomic.Pointer[gorm.DB]
	db.Store(first)
	defer func() { conn_gorm.Close(db.Load()) }()

	// OnChange runs when a reload passes validation and the result differs
	// from the current configuration. Callbacks run in order on the Watch
	// goroutine, and the next reload waits for them to return.
	loader.OnChange(func(old, next *Config) {
		if err := log.Update(next.Log); err != nil {
			log.Error("log config not applied", "error", err)
		}
		// Connections are never rebuilt for you. For a Named field,
		// old.MySQL.Diff(next.MySQL) returns the added, removed and changed
		// names, so only those are rebuilt.
		if reflect.DeepEqual(old.MySQL, next.MySQL) {
			return
		}
		// Open the new connection first: if the new settings are wrong, the
		// old connection keeps serving.
		fresh, err := conn_gorm.OpenMySQL(ctx, next.MySQL)
		if err != nil {
			log.Error("mysql config not applied, keeping the old connection", "error", err)
			return
		}
		stale := db.Swap(fresh)
		// A request may have loaded the old handle just before the swap; give
		// it time to start its query. database/sql's Close then waits for
		// running queries to finish.
		time.AfterFunc(time.Minute, func() { _ = conn_gorm.Close(stale) })
		log.Info("mysql reconnected", "host", next.MySQL.Host)
	})
	// OnError runs when a reload fails at any step: reading a source,
	// parsing, decoding or validating. The loader has already logged the
	// error and keeps the current configuration.
	var reloadFailures atomic.Int64 // e.g. exported as a metric for alerting
	loader.OnError(func(err error) { reloadFailures.Add(1) })
	// Watch reloads 500ms after the last change (config.WithDebounce changes
	// the delay) when:
	//   - a file of File, Profile or DotEnv changes. It watches the directory,
	//     so editors that save by rename and Kubernetes ConfigMap symlink
	//     swaps are seen;
	//   - an etcd key or prefix changes.
	// Environment variables do not change while the process runs. Watch
	// returns nil when ctx is done, and an error when no source can be
	// watched or a watch breaks down.
	go func() {
		if err := loader.Watch(ctx); err != nil {
			log.Error("config watch stopped", "error", err)
		}
	}()

	// loader.Current() returns the latest configuration, shared by every
	// goroutine: read it, never modify it.
	serve(ctx, &db, func() int { return loader.Current().MaxItems })
}

// serve stands for the application. It calls db.Load() and maxItems() once
// per request, so a request sees one version and the next sees the latest.
func serve(ctx context.Context, db *atomic.Pointer[gorm.DB], maxItems func() int) {
	<-ctx.Done()
}

For Redis, close the stale client in its own goroutine (go stale.Close()): go-redis takes about 2s to close a client that never connected.

10 Configuration for each layer

Before, from gin-vue-admin (config.yaml, global/global.go, service/system/sys_user.go, utils/jwt.go):

# config.yaml
jwt:
    signing-key: qmPlus
    expires-time: 7d          # a string, parsed by utils.ParseDuration where the token is made
    buffer-time: 1d
    issuer: qmPlus

// global/global.go
var (
	GVA_DB     *gorm.DB
	GVA_DBList map[string]*gorm.DB
	GVA_REDIS  redis.UniversalClient
	GVA_CONFIG config.Server
	GVA_LOG    *zap.Logger
	// …
)

// service/system/sys_user.go
err = global.GVA_DB.Create(&u).Error

// utils/jwt.go
ep, _ := ParseDuration(global.GVA_CONFIG.JWT.ExpiresTime)

	gin-vue-admin	loadconf
A layer receives	nothing: it reads package global	its own Config and the connections it uses
It depends on	the whole config.Server and every connection	its own types
Its tests	set the globals first	pass values: Config{MaxItems: 2}
A bad value	parsed where it is used; ParseDuration's error is dropped	decoded by the field's type and checked by validate when main loads

With loadconf, changes from example 9: the whole Config is no longer passed around. Each layer declares the settings it needs, and main composes them.

# configs/app.yaml: one section per layer, named by the field main composes it under
mysql:
  host: 127.0.0.1
  user: app
  database: shop
order:
  max_items: 50             # checked by validate:"min=1" when main loads
  timeout: 2s               # decoded as a time.Duration; "2 seconds" fails Load

order/order.go:

// Package order is one layer of the application. It declares the settings
// it needs and does not know where they come from.
package order

import (
	"context"
	"errors"
	"time"

	"gorm.io/gorm"
)

// Config is this layer's part of the configuration. Its defaults and rules
// apply when main loads the whole tree, and errors carry the full path
// (order.max_items: ...). For a rule across fields, add a Validate() error
// method.
type Config struct {
	MaxItems int           `default:"100" validate:"min=1"`
	Timeout  time.Duration `default:"3s"`
}

type Order struct {
	ID    uint
	Items int
}

// Repo receives a connection, not connection settings.
type Repo struct {
	db func() *gorm.DB // a func, so a reconnect (example 9) is picked up
}

func NewRepo(db func() *gorm.DB) *Repo { return &Repo{db: db} }

func (r *Repo) Create(ctx context.Context, o *Order) error {
	return r.db().WithContext(ctx).Create(o).Error
}

// Service receives its settings as a func so that a reload reaches it. A
// setting that never changes at run time can be passed as a plain value.
type Service struct {
	cfg  func() Config
	repo *Repo
}

func NewService(cfg func() Config, repo *Repo) *Service { return &Service{cfg: cfg, repo: repo} }

var ErrTooManyItems = errors.New("too many items")

func (s *Service) Place(ctx context.Context, items int) error {
	cfg := s.cfg() // once per call, so one call sees one version
	if items > cfg.MaxItems {
		return ErrTooManyItems
	}
	ctx, cancel := context.WithTimeout(ctx, cfg.Timeout)
	defer cancel()
	return s.repo.Create(ctx, &Order{Items: items})
}

main.go:

package main

import (
	"context"
	"log"

	"gorm.io/gorm"

	"example.com/app/order"

	"github.com/linzeyan/loadconf/config"
	"github.com/linzeyan/loadconf/connector/conn_gorm"
)

// Config only composes what the layers declare.
type Config struct {
	MySQL config.MySQL
	Order order.Config // keys order.max_items and order.timeout
}

func main() {
	ctx := context.Background()
	loader := config.New[Config](config.From(config.Profile("app"), config.Env("APP")))
	cfg, err := loader.Load(ctx)
	if err != nil {
		log.Fatal(err)
	}
	db, err := conn_gorm.OpenMySQL(ctx, cfg.MySQL)
	if err != nil {
		log.Fatal(err)
	}
	defer conn_gorm.Close(db)

	// Pass each layer its values or connections, never the loader or the
	// whole Config: a layer then depends only on its own types.
	repo := order.NewRepo(func() *gorm.DB { return db })
	svc := order.NewService(func() order.Config { return loader.Current().Order }, repo)
	go loader.Watch(ctx) // handle its error as in example 9

	_ = svc // hand svc to the HTTP or gRPC layer
}

order/order_test.go:

// A layer's tests pass values directly: no config file, no loader.
func TestPlaceRejectsTooManyItems(t *testing.T) {
	s := NewService(func() Config { return Config{MaxItems: 2} }, nil)
	if err := s.Place(context.Background(), 3); err != ErrTooManyItems {
		t.Fatalf("err = %v", err)
	}
}

11 A file path of your own, and struct tags you already have

Before, from etcd (etcd.conf.yml.sample, server/etcdmain/config.go, server/embed/config.go):

# etcd.conf.yml.sample
# Time (in milliseconds) of a heartbeat interval.
heartbeat-interval: 100

# Time (in milliseconds) for an election to timeout.
election-timeout: 1000

# List of comma separated URLs to listen on for peer traffic.
listen-peer-urls: http://localhost:2380

# List of comma separated URLs to listen on for client traffic.
listen-client-urls: http://localhost:2379

# List of this member's client URLs to advertise to the public.
# The URLs needed to be a comma-separated list.
advertise-client-urls: http://localhost:2379

// server/etcdmain/config.go
fs.StringVar(&cfg.configFile, "config-file", "", "Path to the server configuration file. Note that if a configuration file is provided, other command line flags and environment variables will be ignored.")
// …
if cfg.configFile == "" {
	cfg.configFile = os.Getenv(flags.FlagToEnv("ETCD", "config-file"))
}
if cfg.configFile != "" {
	err = cfg.configFromFile(cfg.configFile)
} else {
	err = cfg.configFromCmdLine()
}

// server/embed/config.go
type configJSON struct {
	ListenPeerURLs      string `json:"listen-peer-urls"`
	ListenClientURLs    string `json:"listen-client-urls"`
	AdvertiseClientURLs string `json:"advertise-client-urls"`
	// …
}

func (cfg *configYAML) configFromFile(path string) error {
	b, err := os.ReadFile(path)
	// …
	err = yaml.Unmarshal(b, cfg)
	// …
	if cfg.configJSON.ListenPeerURLs != "" {
		u, err := types.NewURLs(strings.Split(cfg.configJSON.ListenPeerURLs, ","))
		if err != nil {
			fmt.Fprintf(os.Stderr, "unexpected error setting up listen-peer-urls: %v\n", err)
			os.Exit(1)
		}
		cfg.Config.ListenPeerUrls = u
	}
	// … five more copies of this block, one per URL list; the one for
	// advertise-client-urls prints "advertise-peer-urls" on error
}

	etcd	loadconf
Path	--config-file, then $ETCD_CONFIG_FILE	your own flag, passed to config.File
Struct tags	json tags, read through sigs.k8s.io/yaml	the same json tags, with config.WithTagName("json")
URL lists	a comma-separated string, split and parsed after loading, one copied block per list	[]string from a YAML list or a comma-separated string; validate:"dive,url" checks each item
A bad URL	os.Exit(1) inside a function that returns error; the first bad list stops the process	Load returns every bad URL, each with its key

With loadconf, changes from example 1: the path comes from a flag, and the struct keeps the key names its json tags already declare. The file is not touched.

# /etc/etcd/etcd.conf.yml: the same file as before
heartbeat-interval: 100
election-timeout: 1000
listen-peer-urls: http://localhost:2380
listen-client-urls: http://localhost:2379,http://10.0.0.5:2379   # a comma-separated string still works
advertise-client-urls: http://localhost:2379

package main

import (
	"context"
	"flag"
	"log"

	"github.com/linzeyan/loadconf/config"
)

// The json tags name the keys, so the existing file loads as it is. Fields
// without a tag key fall back to snake_case.
type ServerConfig struct {
	HeartbeatInterval   int      `json:"heartbeat-interval" default:"100" validate:"min=1"` // milliseconds
	ElectionTimeout     int      `json:"election-timeout" default:"1000" validate:"min=1"`
	ListenPeerURLs      []string `json:"listen-peer-urls" validate:"min=1,dive,url"`
	ListenClientURLs    []string `json:"listen-client-urls" validate:"min=1,dive,url"`
	AdvertiseClientURLs []string `json:"advertise-client-urls" validate:"dive,url"`
}

func main() {
	path := flag.String("config-file", "/etc/etcd/etcd.conf.yml", "path to the configuration file")
	flag.Parse()

	// A key in the file that matches no field, such as data-dir in etcd's
	// full sample, logs a warning; config.WithStrict() makes it an error.
	cfg, err := config.Load[ServerConfig](context.Background(),
		config.WithTagName("json"),
		config.From(config.File(*path)),
	)
	if err != nil {
		// Every bad item at once, e.g.
		//   config: invalid: listen-peer-urls[1]: does not satisfy "url"
		log.Fatal(err)
	}
	log.Printf("client URLs: %v", cfg.ListenClientURLs) // [http://localhost:2379 http://10.0.0.5:2379]
}

File options:

config.File(path, config.FileFormat("yaml")) // the extension does not say the format: etcd.conf, app.cfg
config.File(path, config.Optional())         // a missing file loads as empty instead of failing
config.WithTagName("yaml")                   // key names from yaml tags instead of config tags

12 A format of your own

Before, from Gitea (custom/conf/app.example.ini, modules/setting/server.go, modules/setting/config_provider.go), which reads app.ini with gopkg.in/ini.v1:

; custom/conf/app.example.ini: the defaults are written here as comments
[server]
;; The protocol the server listens on. One of "http", "https", "http+unix", "fcgi" or "fcgi+unix".
;PROTOCOL = http
;HTTP_ADDR = 0.0.0.0
;; The port to listen on for "http" or "https" protocol. Leave empty when using a unix socket.
;HTTP_PORT = 3000
;CERT_FILE = https/cert.pem
;KEY_FILE = https/key.pem
;GRACEFUL_HAMMER_TIME = 60s

// modules/setting/server.go: one line per key, each with its default again
HTTPAddr = sec.Key("HTTP_ADDR").MustString("0.0.0.0")
HTTPPort = sec.Key("HTTP_PORT").MustString("3000")
protocolCfg := sec.Key("PROTOCOL").String()
switch protocolCfg {
case "", "http":
	Protocol = HTTP
case "https":
	Protocol = HTTPS
	CertFile = sec.Key("CERT_FILE").String()
	KeyFile = sec.Key("KEY_FILE").String()
	// …
}
GracefulHammerTime = sec.Key("GRACEFUL_HAMMER_TIME").MustDuration(60 * time.Second)

// modules/setting/config_provider.go
func (k *iniConfigKey) MustDuration(defaultVal ...time.Duration) time.Duration {
	s := k.String()
	v, err := strconv.ParseInt(s, 10, 64)
	if err == nil {
		d := time.Duration(v) * time.Second
		k.key.SetValue(d.String())
		return d
	}
	d, err := time.ParseDuration(s)
	if err != nil {
		d = util.OptionalArg(defaultVal)
	}
	k.key.SetValue(d.String())
	return d
}

	Gitea	loadconf
Reading	one sec.Key(…).MustXxx(…) line per key, into package variables	one struct; the format only has to produce a tree
Defaults	written twice: as a comment in app.example.ini and as the argument of MustXxx	once, in a default tag
A bad value	GRACEFUL_HAMMER_TIME = 60x becomes 60s without a word	Load fails: server.graceful_hammer_time: invalid duration "60x"
Port	a string	an int, checked by validate

With loadconf, changes from example 1: the format is registered once, then File and Profile accept .ini like .yaml. loadconf does not depend on an INI library; the parser is yours, here built on gopkg.in/ini.v1.

; custom/conf/app.ini: keys match fields case-insensitively, so HTTP_PORT is http_port
[server]
PROTOCOL = https
HTTP_PORT = 3000
CERT_FILE = /etc/gitea/cert.pem
KEY_FILE = /etc/gitea/key.pem
GRACEFUL_HAMMER_TIME = 60s

package main

import (
	"context"
	"log"
	"maps"
	"time"

	"gopkg.in/ini.v1"

	"github.com/linzeyan/loadconf/config"
)

type Config struct {
	Server struct {
		Protocol           string        `default:"http" validate:"oneof=http https"`
		HTTPAddr           string        `default:"0.0.0.0"`
		HTTPPort           int           `default:"3000" validate:"min=1,max=65535"`
		CertFile           string
		KeyFile            string
		GracefulHammerTime time.Duration `default:"60s"`
	}
}

// parseINI returns sections as maps and values as strings. loadconf converts
// the strings to each field's type, as it does for environment variables.
func parseINI(data []byte) (map[string]any, error) {
	f, err := ini.Load(data)
	if err != nil {
		return nil, err
	}
	out := map[string]any{}
	for _, sec := range f.Sections() {
		m := map[string]any{}
		for _, k := range sec.Keys() {
			m[k.Name()] = k.Value()
		}
		if sec.Name() == ini.DefaultSection {
			maps.Copy(out, m) // keys above the first [section]
		} else {
			out[sec.Name()] = m
		}
	}
	return out, nil
}

func main() {
	// Register before the first Load. Registering "yaml" or ".yml" again
	// replaces the built-in parser.
	config.RegisterFormat("ini", parseINI, ".ini")

	cfg, err := config.Load[Config](context.Background(),
		config.From(config.File("custom/conf/app.ini")))
	if err != nil {
		log.Fatal(err)
	}
	log.Printf("listening on %s:%d", cfg.Server.HTTPAddr, cfg.Server.HTTPPort)
}

The same parser serves every source that names a format: config.Bytes("ini", data), and etcd.Key(cli, "/gitea/app.ini") once the extension is registered.

13 Types and checks of your own

Before, from Dex (config.yaml.dist, cmd/dex/config.go, cmd/dex/serve.go):

# config.yaml.dist
web:
  http: 127.0.0.1:5556

  # Uncomment to enable HTTPS endpoint.
  # https: 127.0.0.1:5554
  # tlsCert: /etc/dex/tls.crt
  # tlsKey: /etc/dex/tls.key
  # tlsMinVersion: 1.2
  # tlsMaxVersion: 1.3

# Expiration configuration for tokens, signing keys, etc.
# expiry:
#   deviceRequests: "5m"
#   idTokens: "24h"

// cmd/dex/config.go
type Expiry struct {
	IDTokens       string `json:"idTokens"`
	AuthRequests   string `json:"authRequests"`
	DeviceRequests string `json:"deviceRequests"`
	// …
}

func (c Config) Validate() error {
	checks := []struct {
		bad    bool
		errMsg string
	}{
		{c.Web.HTTP == "" && c.Web.HTTPS == "", "must supply a HTTP/HTTPS  address to listen on"},
		{c.Web.HTTPS != "" && c.Web.TLSCert == "", "no cert specified for HTTPS"},
		{c.Web.HTTPS != "" && c.Web.TLSKey == "", "no private key specified for HTTPS"},
		{c.Web.TLSMinVersion != "" && c.Web.TLSMinVersion != "1.2" && c.Web.TLSMinVersion != "1.3", "supported TLS versions are: 1.2, 1.3"},
		{c.Web.TLSMaxVersion != "" && c.Web.TLSMaxVersion != "1.2" && c.Web.TLSMaxVersion != "1.3", "supported TLS versions are: 1.2, 1.3"},
		{c.Web.TLSMaxVersion != "" && c.Web.TLSMinVersion != "" && c.Web.TLSMinVersion > c.Web.TLSMaxVersion, "TLSMinVersion greater than TLSMaxVersion"},
		// … the same TLS checks again for gRPC
	}
	// …
}

// cmd/dex/serve.go: durations are parsed where they are used, after
// storage has been opened
idTokensValidFor := 24 * time.Hour // default
if c.Expiry.IDTokens != "" {
	var err error
	idTokensValidFor, err = time.ParseDuration(c.Expiry.IDTokens)
	if err != nil {
		return fmt.Errorf("invalid config value %q for id token expiry: %v", c.Expiry.IDTokens, err)
	}
}
// … the same block for AuthRequests and DeviceRequests

allowedTLSVersions := map[string]int{ // the version list again, to convert
	"1.2": tls.VersionTLS12,
	"1.3": tls.VersionTLS13,
}

	Dex	loadconf
Durations	strings, parsed where they are used; the parse-and-wrap block is written three times	time.Duration fields, decoded by Load
A bad duration	reported after storage has been opened	Load fails before anything is opened
Defaults	24 * time.Hour // default in serve.go; the struct does not show it	default:"24h" on the field
TLS versions	accepted as strings in Validate, converted by a second list in serve.go	one type: UnmarshalText accepts and converts
Error messages	no key: supported TLS versions are: 1.2, 1.3 stands for four fields	each error carries its key path

With loadconf, changes from example 1: a field type decodes itself, validate tags hold the rules about one value, and a Validate method holds the rules that relate several.

# configs/app.yaml
web:
  http: 127.0.0.1:5556
  https: 127.0.0.1:5554
  tls_cert: /etc/dex/tls.crt
  tls_key: /etc/dex/tls.key
  tls_min_version: 1.2       # decoded by TLSVersion.UnmarshalText
expiry:
  id_tokens: 24h
  device_requests: 5m

package main

import (
	"context"
	"crypto/tls"
	"errors"
	"fmt"
	"log"
	"time"

	"github.com/linzeyan/loadconf/config"
)

// TLSVersion decodes "1.2" or "1.3" into crypto/tls's constant: the list of
// accepted versions and the conversion are one piece of code. A type that
// implements encoding.TextUnmarshaler decodes from any source's strings.
type TLSVersion uint16

func (v *TLSVersion) UnmarshalText(b []byte) error {
	switch string(b) {
	case "1.2":
		*v = tls.VersionTLS12
	case "1.3":
		*v = tls.VersionTLS13
	default:
		return fmt.Errorf("unsupported TLS version %q (want 1.2 or 1.3)", b)
	}
	return nil
}

type Web struct {
	HTTP          string     `validate:"omitempty,hostname_port"`
	HTTPS         string     `validate:"omitempty,hostname_port"`
	TLSCert       string
	TLSKey        string
	TLSMinVersion TLSVersion `default:"1.2"`
	TLSMaxVersion TLSVersion // 0 when unset: Go's default
}

// Validate holds the rules across fields. It runs after decoding and after
// the validate tags; each error it returns is reported under this struct's
// key, web.
func (w Web) Validate() error {
	var errs []error
	if w.HTTP == "" && w.HTTPS == "" {
		errs = append(errs, errors.New("set http, https or both"))
	}
	if w.HTTPS != "" && (w.TLSCert == "" || w.TLSKey == "") {
		errs = append(errs, errors.New("https needs tls_cert and tls_key"))
	}
	if w.TLSMaxVersion != 0 && w.TLSMinVersion > w.TLSMaxVersion {
		errs = append(errs, errors.New("tls_min_version is above tls_max_version"))
	}
	return errors.Join(errs...)
}

type Expiry struct {
	IDTokens       time.Duration `default:"24h" validate:"min=1m"`
	AuthRequests   time.Duration `default:"24h" validate:"min=1m"`
	DeviceRequests time.Duration `default:"5m" validate:"min=1m"`
}

type Config struct {
	Web    Web
	Expiry Expiry
}

func main() {
	cfg, err := config.Load[Config](context.Background(), config.From(config.Profile("app")))
	if err != nil {
		// Decoding comes first; each stage reports all its problems at once,
		// each with its key path:
		//   config: decode: web.tls_min_version: unsupported TLS version "1.1" (want 1.2 or 1.3)
		// Then the rules, once every value has decoded:
		//   config: invalid: web: https needs tls_cert and tls_key
		//   expiry.id_tokens: does not satisfy "min=1m"
		log.Fatal(err)
	}
	log.Printf("id tokens valid for %s", cfg.Expiry.IDTokens)
}

Rules that need code go in methods of the type they concern:

Method	Runs	For
UnmarshalText([]byte) error	while decoding a string	a value written as one string: versions, levels, addresses (netip.Addr already has it)
UnmarshalConfig(any) error	while decoding, with the raw value: map, list, string, number	a value written in several shapes, e.g. "5s" or {min: 1s, max: 5s}
SetDefaults()	before decoding, after the default tags	defaults that are computed, e.g. from the host name
Validate() error	after decoding and the validate tags	rules across fields

The validate rules are listed under Struct tags.

14 A source of your own

Before, from CDS (docs/content/hosting/configuration.md, engine/config.go, engine/config_vault.go). The whole TOML file is stored as one Consul key or one Vault secret:

# docs/content/hosting/configuration.md
$ consul kv put cds/config.api.toml -
<PASTE YOUR CONFIGURATION>
<ENDS WITH CTRL-D>
Success! Data written to: cds/config.api.toml

$ $PATH_TO_CDS/engine start api --remote-config localhost:8500 --remote-config-key cds/config.api.toml

# or
$ myConfig=`cat conf.toml`
$ vault write secret/cds/conf data=$myConfig
$ $PATH_TO_CDS/engine start api --vault-addr=http://myvault.com  --vault-token=XXXX

// engine/config.go: one source out of three
switch {
case remoteCfg != "":
	viper.AddRemoteProvider("consul", remoteCfg, remoteCfgKey)
	viper.SetConfigType("toml")
	if err := viper.ReadRemoteConfig(); err != nil {
		sdk.Exit(err.Error())
	}
case vaultAddr != "" && vaultToken != "":
	// I hope one day vault will be a standard viper remote provider
	s, err := VaultNewSecret(vaultToken, vaultAddr)
	// …
	cfgFileContent, err := s.GetFromVault(vaultConfKey) // "/secret/cds/conf"
	// …
	viper.SetConfigType("toml")
	if err := viper.ReadConfig(bytes.NewBufferString(cfgFileContent)); err != nil {
		sdk.Exit("Unable to read config: %v", err)
	}
// … or a local file
}

// engine/config_vault.go
func (secret *VaultSecret) GetFromVault(s string) (string, error) {
	conf, err := secret.Client.Logical().Read(s)
	if err != nil {
		return "", sdk.WithStack(err)
	} else if conf == nil {
		log.Warn(context.Background(), "vault> no value found at %q", s)
		return "", nil
	}
	// …
}

	CDS	loadconf
Sources	one of Consul, Vault or a file, picked by a switch	layered: a file, then Consul, then variables, merged key by key
A missing key	Vault: a warning, then an empty configuration	Load fails, naming the source
Format and path	TOML, and the Vault path /secret/cds/conf, fixed in code	fields of the source
Errors	sdk.Exit at every step	Load returns them
Token	--vault-token=XXXX on the command line, where ps shows it	the variable CONSUL_HTTP_TOKEN, sent as a header

With loadconf, changes from example 9: a type with Load and String is a source; with Watch as well, it takes part in hot reload. This one reads a Consul key through Consul's HTTP API, so it needs no Consul client.

# The document in Consul holds what operators change at run time.
consul kv put order-service/prod.toml - <<'EOF'
max_items = 50
[log]
level = "debug"
EOF

package main

import (
	"context"
	"fmt"
	"io"
	"log"
	"net/http"
	"os"

	"github.com/linzeyan/loadconf/config"
)

// consulKey loads one Consul KV key holding a whole document.
type consulKey struct {
	addr   string // e.g. http://127.0.0.1:8500
	key    string // e.g. order-service/prod.toml
	format string // "toml": the key's content, parsed by config.Parse
	token  string // an ACL token, or ""
}

func (c *consulKey) String() string { return "consul(" + c.key + ")" }

func (c *consulKey) Load(ctx context.Context) (map[string]any, error) {
	data, _, err := c.get(ctx, "")
	if err != nil {
		return nil, err
	}
	return config.Parse(c.format, data)
}

// Watch blocks until ctx is done. Each request is a Consul blocking query: it
// returns when the key's index moves past the last one, or after five
// minutes with the same index. A spurious call to changed is harmless: the
// loader compares the reloaded configuration with the current one.
func (c *consulKey) Watch(ctx context.Context, changed func()) error {
	_, index, err := c.get(ctx, "")
	for err == nil {
		var next string
		if _, next, err = c.get(ctx, index); err == nil && next != index {
			index = next
			changed()
		}
	}
	if ctx.Err() != nil {
		return nil
	}
	return err
}

func (c *consulKey) get(ctx context.Context, index string) ([]byte, string, error) {
	url := c.addr + "/v1/kv/" + c.key + "?raw"
	if index != "" {
		url += "&wait=5m&index=" + index
	}
	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
	if err != nil {
		return nil, "", err
	}
	if c.token != "" {
		req.Header.Set("X-Consul-Token", c.token)
	}
	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		return nil, "", err
	}
	defer resp.Body.Close()
	if resp.StatusCode != http.StatusOK { // 404: the key does not exist
		return nil, "", fmt.Errorf("GET /v1/kv/%s: %s", c.key, resp.Status)
	}
	data, err := io.ReadAll(resp.Body)
	return data, resp.Header.Get("X-Consul-Index"), err
}

type Config struct {
	Log      config.Log
	MaxItems int `default:"100"`
}

func main() {
	ctx := context.Background()
	loader := config.New[Config](config.From(
		config.Profile("app"), // the settings in version control
		&consulKey{
			addr:   "http://127.0.0.1:8500",
			key:    "order-service/prod.toml",
			format: "toml",
			token:  os.Getenv("CONSUL_HTTP_TOKEN"),
		},
		config.Env("APP"), // secrets
	))
	cfg, err := loader.Load(ctx)
	if err != nil {
		// e.g. config: load consul(order-service/prod.toml): GET /v1/kv/order-service/prod.toml: 404 Not Found
		log.Fatal(err)
	}
	log.Printf("max items: %d", cfg.MaxItems)
	// `consul kv put` on the key reloads the configuration, as a file
	// change does (example 9).
	go loader.Watch(ctx)
	// …
}

Built-in sources for documents that do not live in files:

//go:embed defaults.yaml
var defaults []byte

config.From(
	config.Bytes("yaml", defaults),                  // defaults compiled into the binary
	config.File("/etc/app/app.yaml", config.Optional()),
)

// In a test: a configuration without any file
cfg, err := config.Load[Config](ctx, config.From(config.Map(map[string]any{
	"max_items": 2,
	"log":       map[string]any{"level": "debug"},
})))

Reference

Sources and precedence

cfg, err := config.Load[Config](ctx, config.From(
	config.Profile("app"),                                 // configs/app.yaml + configs/app.$ENV.yaml
	config.File("/etc/app/extra.toml", config.Optional()), // .json .yaml .yml .toml, or a registered format; Optional: may be missing
	config.DotEnv(".env", "APP", config.DotEnvOptional()), // same names as Env
	etcd.Prefix(cli, "/order-service/prod/"),              // /order-service/prod/mysql/main/host -> mysql.main.host
	config.Env("APP"),                                     // real environment variables
))
// A later source overrides an earlier one; maps merge key by key and Named
// instances merge by name. Fields with an env tag are set after all sources.
// Also available: config.Bytes("yaml", data) and config.Map(m), for tests and
// defaults built in code. A custom source implements config.Source, and
// config.WatchableSource to take part in hot reload (example 14).
// config.RegisterFormat("ini", parse, ".ini") adds a format (example 12);
// config.FileFormat("yaml") reads a file whose extension does not name one.

Which sources read ENV

Only Profile reads ENV. Every other source reads the path, prefix or key it is given; to vary it by environment, put the environment into that path or key.

Source	Reads ENV	Varies by environment through
config.Profile("app")	yes: app.yaml, then app.$ENV.yaml; with $CONFIG_PATH set, that file only	file name
config.File(path)	no	the path you build
config.Env("APP"), config.DotEnv	no	the values each deployment sets
etcd.Key, etcd.Prefix	no	the key: "/order-service/"+env+"/"
env tag	no	the values each deployment sets

Environment variable names

# config.Env("APP") reads variables starting with APP_. "__" separates levels;
# a single "_" stays in the key. Change the separator with config.EnvSeparator.
APP_PORT=9090                          # port
APP_MYSQL__MAIN__PASSWORD=s3cret       # mysql.main.password
APP_MYSQL__MAIN__CONN_MAX_LIFETIME=10m # mysql.main.conn_max_lifetime
APP_REDIS__ADDRS=r1:6379,r2:6379       # a list: comma separated
APP_MYSQL__REPORT__HOST=10.0.0.3       # adds the instance "report"
APP_MYSQL__MAIN__PARAMS__PARSETIME=1   # map keys arrive lower-cased: parsetime
# An empty value counts as unset. A variable that matches no field is ignored
# without a warning, even under WithStrict: environments hold many unrelated
# variables.

Struct tags

type Config struct {
	Port    int             `default:"8080" env:"PORT"` // env: read $PORT after all sources
	Timeout time.Duration   `default:"5s"`              // durations are strings: "500ms", "5m"
	Buffer  config.ByteSize `default:"64MiB"`           // 64MiB = 64×1024², 64MB = 64×1000²
	Hosts   []string        `validate:"min=1,dive,hostname_port"` // rules: see below
	Token   config.Secret   // prints as ******; Token.Value() is the real string
	GRPC    GRPCConfig      `env-prefix:"GRPC_"`        // prefix for the env tags inside GRPCConfig
	Legacy  string          `config:"legacy_name"`      // explicit key; "-" skips the field
}
// A type can hook into loading by implementing SetDefaults(), Validate() error
// or UnmarshalConfig(any) error (example 13). encoding.TextUnmarshaler types,
// such as netip.Addr, decode from strings. config.WithTagName("yaml") reads
// yaml tags instead of config tags (example 11).

validate holds rules separated by commas, checked after decoding. A field reports the first rule it fails. An unknown rule, or a parameter that does not fit the field's type, fails Load whatever the value, so a typo cannot turn a check off.

Rule	Passes when
required	the value is set: not zero, not empty, not a nil pointer
omitempty	always; when the value is empty, the rules after it are skipped
min=n, max=n, len=n, gt=n, gte=n, lt=n, lte=n	strings (in characters), slices and maps compare their length; numbers, time.Duration and config.ByteSize compare their value, so min=1s and max=64MiB work. gte is min, lte is max
oneof=a b c	the value is one of the values separated by spaces
url	an absolute URL with a scheme and a host or path: https://host/path, file:///tmp/x
hostname_port	host:port or :port, with a port from 1 to 65535
ip, cidr	an IPv4 or IPv6 address; a network such as 10.0.0.0/8
dive	the rules after it apply to each element of a slice or map

A Secret is checked by the string it holds. Fields of structs inside slices, maps and Named are checked without dive. Rules that need code, such as one field depending on another, go in a Validate() error method.

Open and New

rdb, err := conn_redis.Open(ctx, cfg.Redis) // build, then check within ping_timeout; on failure, close and return the error
rdb, err := conn_redis.New(cfg.Redis)       // build only, no network I/O; connects on first use
o, err := conn_redis.Options(cfg.Redis)     // the driver's own settings, to build a client yourself

A connector has no New when its driver cannot build a client without connecting, and no Open when it is an HTTP API client with no connection to check.

Connector	Open	New	Driver settings
conn_sql	OpenMySQL, OpenPostgres	NewMySQL, NewPostgres	MySQLConfig, PostgresConfig, PostgresConnString
conn_mssql	Open	New	Config, Connector
conn_oracle	Open	New	URL, Connector
conn_sqlite	Open	New	DSN
conn_trino	Open	New	DSN
conn_clickhouse	Open (native API), OpenDB (database/sql)	New, NewDB	Options, Connector
conn_gorm	OpenMySQL, OpenPostgres	none: gorm.Open may connect while initializing the dialector (MySQL queries the server version)
conn_gorm_mssql, conn_gorm_sqlite	Open	none: as conn_gorm
conn_redis	Open	New	Options
conn_mongo	Open, OpenDatabase	New	ClientOptions
conn_cassandra, conn_scylla	Open	none: gocql connects when it creates a session	ClusterConfig
conn_es8, conn_es9	Open	New, NewTypedClient	Config
conn_opensearch	Open	New	Config
conn_kafka_franz	Open	New	Options
conn_kafka_sarama	OpenClient, OpenSyncProducer, OpenAsyncProducer, OpenConsumerGroup, OpenConsumer, OpenClusterAdmin	none: sarama connects when it builds a client	Config
conn_mqtt	Open	New	ClientOptions
conn_mqtt_rest	none: HTTP API	New, NewEMQX, NewGeneric
conn_streamload	none: HTTP API	New

// Every entry point takes options for what a config file cannot hold.
db, err := conn_gorm.OpenMySQL(ctx, cfg.MySQL.MustGet("main"),
	conn_gorm.WithGormConfig(&gorm.Config{PrepareStmt: true}))

// gocql and sarama cannot be interrupted while connecting. When ctx ends
// first, Open returns ctx.Err() at once and closes the session or client
// that arrives later. Give ctx a deadline, or bound the driver itself with
// options.connect_timeout (gocql) or options.net.dial_timeout (sarama).
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
session, err := conn_cassandra.Open(ctx, cfg.Cassandra)

Connectors

SQL

import (
	"github.com/linzeyan/loadconf/connector/conn_gorm"
	"github.com/linzeyan/loadconf/connector/conn_gorm_mssql"
	"github.com/linzeyan/loadconf/connector/conn_gorm_sqlite"
	"github.com/linzeyan/loadconf/connector/conn_mssql"
	"github.com/linzeyan/loadconf/connector/conn_oracle"
	"github.com/linzeyan/loadconf/connector/conn_sql"
	"github.com/linzeyan/loadconf/connector/conn_sqlite"
)

sqlDB, err := conn_sql.OpenMySQL(ctx, cfg.MySQL.MustGet("main"))       // *sql.DB
pg, err := conn_sql.OpenPostgres(ctx, cfg.Postgres.MustGet("main"))    // *sql.DB through pgx
gdb, err := conn_gorm.OpenMySQL(ctx, cfg.MySQL.MustGet("main"))        // *gorm.DB; turns parseTime on unless set
gpg, err := conn_gorm.OpenPostgres(ctx, cfg.Postgres.MustGet("main"))
mss, err := conn_mssql.Open(ctx, cfg.SQLServer.MustGet("erp"))
gmss, err := conn_gorm_mssql.Open(ctx, cfg.SQLServer.MustGet("erp"))
ora, err := conn_oracle.Open(ctx, cfg.Oracle.MustGet("core"))          // pure Go: no Instant Client
lite, err := conn_sqlite.Open(ctx, cfg.SQLite.MustGet("local"))        // no cgo
glite, err := conn_gorm_sqlite.Open(ctx, cfg.SQLite.MustGet("local"))

# config.MySQL also serves TiDB, OceanBase, and Doris and StarRocks (the FE's
# port 9030). config.Postgres also serves CockroachDB, YugabyteDB, Redshift,
# TimescaleDB and Greenplum.
mysql:
  main:
    dsn: "app@tcp(db1:3306)/shop"   # or host, port, user, database
    password: s3cret                # fields override the DSN, e.g. a password from a secret store
    params: {loc: Local}
    max_open_conns: 20
sqlserver:
  erp: {host: mssql, user: sa, password: pw, database: erp, params: {encrypt: "true"}}
oracle:
  core: {host: ora, service: FREEPDB1, user: app, password: pw}
sqlite:
  local: {path: data/app.db}        # or ":memory:"

NoSQL

import (
	"github.com/linzeyan/loadconf/connector/conn_cassandra"
	"github.com/linzeyan/loadconf/connector/conn_mongo"
	"github.com/linzeyan/loadconf/connector/conn_redis"
	"github.com/linzeyan/loadconf/connector/conn_scylla"
)

rdb, err := conn_redis.Open(ctx, cfg.Redis)          // redis.UniversalClient: single, cluster or sentinel
mdb, err := conn_mongo.OpenDatabase(ctx, cfg.Mongo)  // *mongo.Database; conn_mongo.Open returns the client
cass, err := conn_cassandra.Open(ctx, cfg.Cassandra) // *gocql.Session
scy, err := conn_scylla.Open(ctx, cfg.Scylla)        // shard-aware with the replace directive below

redis:
  addrs: ["r1:6379", "r2:6379", "r3:6379"]  # several addresses: cluster; with master_name: sentinel
  password: s3cret
mongo:
  uri: "mongodb://m1,m2/?replicaSet=rs0"
  database: shop
scylla:
  hosts: [s1, s2]
  local_dc: dc1
  options: {consistency: LOCAL_QUORUM}      # gocql expects capitals

// go.mod of an application that uses conn_scylla. Without this line the
// build succeeds and connects through upstream gocql: no shard awareness,
// and Open logs a warning.
replace github.com/gocql/gocql => github.com/scylladb/gocql v1.19.0

OLAP

import (
	"github.com/linzeyan/loadconf/connector/conn_clickhouse"
	"github.com/linzeyan/loadconf/connector/conn_streamload"
	"github.com/linzeyan/loadconf/connector/conn_trino"
)

ch, err := conn_clickhouse.Open(ctx, cfg.ClickHouse)     // native API: fastest for batch inserts
chDB, err := conn_clickhouse.OpenDB(ctx, cfg.ClickHouse) // database/sql
tr, err := conn_trino.Open(ctx, cfg.Trino)               // checked with SELECT 1

// Bulk loads into Doris or StarRocks: Stream Load rather than INSERT.
// A failed request is retried on the next FE under the same label, and the
// server deduplicates by label, so a batch is loaded at most once. Data
// errors are not retried: they return a *conn_streamload.LoadError with
// ErrorURL.
sl, err := conn_streamload.New(cfg.Doris) // cfg.Doris is a config.StreamLoad

res, err := sl.LoadJSON(ctx, "events", rows) // rows: a slice, sent as one JSON array
res, err = sl.LoadCSV(ctx, "events", records, conn_streamload.LoadOptions{
	Headers: map[string]string{"columns": "id,name"},
})

w, err := sl.NewWriter("events") // flushes at max_rows, max_bytes or flush_interval
err = w.WriteJSON(ctx, event)
err = w.Close(ctx)

// Two-phase commit. Doris accepts one Load per transaction, StarRocks
// several. A retry resends the data, so pass a seekable reader
// (*bytes.Reader, *os.File).
tx, err := sl.Begin(ctx, "orders")
if _, err := tx.Load(ctx, bytes.NewReader(data)); err != nil {
	_ = tx.Abort(ctx)
} else {
	err = tx.Commit(ctx)
}

doris:
  flavor: doris             # or starrocks
  addrs: ["fe1:8030", "fe2:8030"]
  database: dw
  password: pw
  headers: {strict_mode: "true"}

Search

import (
	"github.com/linzeyan/loadconf/connector/conn_es8" // or conn_es9
	"github.com/linzeyan/loadconf/connector/conn_opensearch"
)

es, err := conn_es8.Open(ctx, cfg.Elasticsearch)
typed, err := conn_es8.NewTypedClient(cfg.Elasticsearch)
osc, err := conn_opensearch.Open(ctx, cfg.OpenSearch)

elasticsearch:
  addresses: ["https://es1:9200"]
  api_key: <key>            # or username and password, service_token, cloud_id
  options: {max_retries: 5, compress_request_body: true}

Messaging

import (
	"github.com/twmb/franz-go/pkg/kgo"

	"github.com/linzeyan/loadconf/connector/conn_kafka_franz"
	"github.com/linzeyan/loadconf/connector/conn_kafka_sarama"
	"github.com/linzeyan/loadconf/connector/conn_mqtt"
	"github.com/linzeyan/loadconf/connector/conn_mqtt_rest"
)

// franz-go: one client produces and consumes; its settings are kgo options.
kc, err := conn_kafka_franz.Open(ctx, cfg.Kafka,
	kgo.DefaultProduceTopic("events"), kgo.ProducerBatchCompression(kgo.ZstdCompression()))

// sarama: settings go in options. The producer is idempotent and waits for
// all in-sync replicas by default.
prod, err := conn_kafka_sarama.OpenSyncProducer(ctx, cfg.Kafka)
group, err := conn_kafka_sarama.OpenConsumerGroup(ctx, cfg.Kafka)
err = group.Consume(ctx, cfg.Kafka.Consumer.Topics, handler)

// MQTT 3.1.1 through Paho. With options.connect_retry set, Open keeps
// retrying: give ctx a deadline.
mc, err := conn_mqtt.Open(ctx, cfg.MQTT)

// EMQX REST API, versions 5.8 to 6.3.
emqx, err := conn_mqtt_rest.NewEMQX(cfg.EMQX)
res, err := emqx.Publish(ctx, conn_mqtt_rest.Message{Topic: "a/b", Payload: []byte("hi"), QoS: 1})
// res.NoSubscribers: published, but nobody was subscribed

kafka:
  brokers: ["k1:9092"]
  consumer: {group: order-service, topics: [orders], initial_offset: earliest}
mqtt:
  brokers: ["tcp://emqx:1883"]
  client_id: order-service
  client_id_random_suffix: true   # replicas of one service must not share a client ID, or they disconnect each other
emqx:
  base_url: http://emqx:18083
  auth: {username: <api-key>, password: <api-secret>}

Logger

import (
	"github.com/linzeyan/loadconf/logger"
	"github.com/linzeyan/loadconf/logger/logger_zap"
	"github.com/linzeyan/loadconf/logger/logger_zerolog"
)

log, err := logger.New(ctx, cfg.Log) // embeds *slog.Logger
defer log.Close()                    // flushes the queues of the elasticsearch and otlp outputs
log.SetDefault()
// Connections opened after SetDefault log through log:
//   gorm: failed queries, and queries slower than 200ms
//   MySQL, PostgreSQL: warnings and above
//   MongoDB, Cassandra/Scylla, ClickHouse, Kafka (franz-go), Stream Load
//   Elasticsearch, OpenSearch: every request; 4xx and 5xx as warnings

zl, err := logger_zap.New(ctx, cfg.Log)     // the same configuration and outputs, as zap
zr, err := logger_zerolog.New(ctx, cfg.Log) // or as zerolog

log:
  level: info                 # trace, debug, info, warn, error; hot reload applies changes
  format: json                # or text: stdout, stderr and file only
  add_source: true
  service: order-service      # adds a service field
  fields: {env: prod}         # static fields on every record
  outputs:                    # stdout when empty
    stdout: {}
    file: {path: /var/log/app/app.log, max_size: 100MiB, max_backups: 7, max_age: 72h, compress: true}
    errors: {type: file, path: /var/log/app/error.log, level: error}
    gelf: {addr: "graylog:12201"}
    syslog: {network: udp, addr: "10.0.0.5:514"}
    elasticsearch: {addresses: ["https://es:9200"], index: "logs-{service}-{date:2006.01.02}"}

import (
	"github.com/linzeyan/loadconf/connector/conn_gorm_log"
	"github.com/linzeyan/loadconf/logger/sink_otlp"
)

// OTLP is a separate module because of its gRPC dependencies; pass it in.
log, err := logger.New(ctx, cfg.Log,
	logger.WithSink("otlp", sink_otlp.Open),
	logger.WithContextAttrs(sink_otlp.TraceAttrs)) // adds trace_id and span_id from ctx
log.InfoContext(ctx, "handled")

// A driver logger other than the default is passed with WithLogger. gorm's
// levels can live in the config file:
type Config struct {
	GormLog conn_gorm_log.Config // slow_threshold, level, parameterized_queries
}
gdb, err := conn_gorm.OpenMySQL(ctx, cfg.MySQL.MustGet("main"),
	conn_gorm.WithGormConfig(&gorm.Config{Logger: conn_gorm_log.New(nil, cfg.GormLog)}))
es, err := conn_es8.Open(ctx, cfg.Elasticsearch, conn_es8.WithLogger(auditLog))

// These drivers only have a process-wide logger; set it once.
redis.SetLogger(conn_redis.SlogLogger(log.Logger, slog.LevelWarn))
sarama.Logger = conn_kafka_sarama.SlogLogger(log.Logger, slog.LevelInfo)
mssql.SetContextLogger(conn_mssql.SlogLogger(log.Logger))
restore := conn_mqtt.SetLogger(log.Logger)

Versioning

# Every module is released under the same version; upgrade them together.
go get github.com/linzeyan/loadconf/config@v0.3.0 github.com/linzeyan/loadconf/connector/conn_gorm@v0.3.0
# Mixed versions compile, but Go selects the highest version any module
# requires of a shared module such as config, so an older pin does not hold.

Development, testing and releases: CONTRIBUTING.md.