Configuration

This page documents how to configure Apache Iceberg Go: the CLI's YAML config file, per-catalog Option surfaces, file-system credentials, table write properties, concurrency, and how to plug in custom catalogs and IO backends.

Only properties and options that the iceberg-go code actually reads are listed. Properties defined in the Apache Iceberg spec but not yet wired into iceberg-go are intentionally omitted - check pkg.go.dev/github.com/apache/iceberg-go for the latest read sites.

CLI configuration file

The iceberg CLI loads catalog defaults from ~/.iceberg-go.yaml (override the directory with GOICEBERG_HOME). The schema, defined in config/config.go, is:

default-catalog: default
max-workers: 5
catalog:
  default:
    type: rest
    uri: https://example.com/iceberg
    warehouse: s3://my-bucket/warehouse
    credential: <client-id>:<client-secret>
    output: text
    aws-profile: ""
    rest:
      sigv4-enabled: false
      signing-name: ""
      signing-region: ""

Catalog options

Each catalog package exposes its own functional Option set. The lists below reflect the public option surface; pkg.go.dev is authoritative for the current set.

REST (catalog/rest)

The most option-rich surface. Source: catalog/rest/options.go.

AWS SigV4 is an optional backend. catalog/rest no longer links the AWS SDK. To sign REST requests with SigV4, pull in the catalog/rest/sigv4 sub-package one of two ways:

• Property or ambient credentials: add a blank import _ "github.com/apache/iceberg-go/catalog/rest/sigv4" and enable signing with WithSigV4 / WithSigV4RegionSvc or the rest.sigv4-enabled property. The backend signs with the s3.* credential properties when set (falling back to the rest.* aliases), otherwise with the AWS default credential chain.
• Explicit config: pass sigv4.WithAWSConfig(cfg) to NewCatalog alongside WithSigV4 / WithSigV4RegionSvc (no blank import needed). It supplies the aws.Config; the signing service comes from those options (or a server /v1/config override), and so does the region when one is set. Otherwise the region falls back to cfg.Region.

The former rest.WithAwsConfig(aws.Config) has been removed in favor of these paths. Because signing is enabled by WithSigV4 / WithSigV4RegionSvc (or the rest.sigv4-enabled property) rather than by supplying a config, a program that loads a REST catalog with rest.sigv4-enabled=true now fails at startup until the catalog/rest/sigv4 backend is blank-imported; the error names the import to add.

Metrics reporting

The REST catalog can POST scan and commit metrics to the catalog's .../tables/{table}/metrics endpoint. These properties are passed as client-supplied catalog properties.

Note on cross-client parity. Iceberg Java defaults metrics reporting to on (METRICS_REPORTING_ENABLED_DEFAULT=true); iceberg-go deliberately defaults it off, so no client emits new outbound telemetry unless it explicitly opts in. A Java client that never set the flag will therefore stop reporting when ported here until rest-metrics-reporting-enabled=true is set. Enablement is read from client properties only — a server cannot turn on reporting the client never asked for — but an explicit server override setting the flag to false is honored (disabling is always safe), so an operator can suppress reporting fleet-wide. rest-metrics-reporting-timeout-ms is a Go-specific extension with no Java, PyIceberg, or Rust equivalent; do not assume parity on it in multi-client deployments.

Hive (catalog/hive)

Source: catalog/hive/options.go.

• WithURI(uri string) - Thrift URI for the Hive Metastore (e.g. thrift://127.0.0.1:9083).
• WithWarehouse(warehouse string)
• WithProperties(props iceberg.Properties)

Lock-check catalog properties (passed via WithProperties / catalog.Load props):

Invalid values (non-positive waits/retries, or min ≥ max) are ignored and the previous/default settings are kept.

Glue (catalog/glue)

Source: catalog/glue/options.go.

• WithAWSConfig(cfg aws.Config) - AWS SDK v2 config; respects the AWS default credential chain.
• WithAWSProperties(props AWSProperties) - catalog properties, such as glue.id and the s3.* FileIO keys. NewCatalog ignores the glue.region, endpoint, credential, and retry keys; only catalog.Load reads them.

SQL (catalog/sql)

The SQL catalog has no functional-option surface. Construct it with NewCatalog:

db, _ := sql.Open(sqliteshim.ShimName, "file:catalog.db")
cat, err := sqlcat.NewCatalog("default", db, sqlcat.SQLite, iceberg.Properties{
    "warehouse": "file:///tmp/warehouse",
})

Supported dialects: sqlcat.Postgres, sqlcat.MySQL, sqlcat.SQLite, sqlcat.MSSQL, sqlcat.Oracle (catalog/sql/sql.go:50).

Shared options on the base catalog package

Operations that create or update tables/views accept these (catalog/catalog.go):

• WithLocation, WithPartitionSpec, WithSortOrder, WithProperties, WithStagedUpdates
• View-specific: WithViewLocation, WithViewProperties

File-system credentials

iceberg-go registers the local file system (file://) automatically. Cloud schemes are not registered until you add a blank import:

import _ "github.com/apache/iceberg-go/io/gocloud/s3"    // s3, s3a, s3n, oss
import _ "github.com/apache/iceberg-go/io/gocloud/gcs"   // gs
import _ "github.com/apache/iceberg-go/io/gocloud/azure" // abfs, abfss, wasb, wasbs

Each backend package links its own cloud SDK. Importing io/gocloud registers all three at once and links all cloud SDKs.

Without a matching blank import, these schemes return ErrIOSchemeNotFound with a hint naming the package to import.

All credential and tuning property keys are constants in io/config.go. They can be supplied through table properties, catalog properties, or per-call iceberg.Properties arguments depending on context.

S3

Authentication is resolved in this order (io/gocloud/s3/s3.go):

1. Static credentials in properties: s3.access-key-id + s3.secret-access-key (+ optional s3.session-token).
2. The standard AWS SDK v2 default credential chain - environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN), ~/.aws/credentials, container/IAM role.

Tuning properties:

Google Cloud Storage

Authentication resolution (io/gocloud/gcs/gcs.go):

1. Explicit JSON key bytes via gcs.jsonkey or path via gcs.keypath.
2. Optional gcs.credtype selecting one of service_account, authorized_user, impersonated_service_account, external_account.
3. The GCP default credentials chain (gcp.DefaultCredentials) - falls back to anonymous if no creds are found.

Tuning properties:

Azure Data Lake Storage / Blob

Authentication is selected in the following order (io/gocloud/azure/azure.go); the first applicable option wins, and if none of 1-5 apply the default credential chain (6) is used:

1. Shared key: a nonempty adls.auth.shared-key.account.name selects this path and requires a nonempty adls.auth.shared-key.account.key.
2. Per-host SAS token: adls.sas-token.<hostname>, where <hostname> exactly matches the storage account hostname (for example, myaccount.dfs.core.windows.net).
3. Per-account connection string: adls.connection-string.<account-name> (for example, adls.connection-string.myaccount).
4. Static access token: a nonempty adls.token, supplied without the Bearer prefix.
5. Managed identity: adls.auth.managed-identity.enabled set to exactly "true" selects this path and uses ManagedIdentityCredential directly. Set adls.client-id to select a user-assigned managed identity; otherwise, the system-assigned managed identity is used.
6. Default credential chain: DefaultAzureCredential, which includes managed identity among its credential sources.

adls.token accepts an OAuth2 access token for Azure Storage, including a token supplied by a catalog. An empty value is ignored. A rejected or expired token does not fall back to managed identity or the default credential chain.

The token is static and is not refreshed. As in Java and PyIceberg, the credential reports a one-hour cache window to the Azure SDK each time the SDK requests the token. This does not extend the token's actual validity: Azure enforces its expiration. Callers must obtain a new token and recreate the FileIO when it expires; changing the original properties map does not update an existing FileIO.

A REST catalog can also vend adls.token as a storage credential. The token carries no expiry of its own, so a vended token's expiry comes from expiration-time in the same credential when the catalog supplies one. Without it, a table FileIO that can fetch new credentials assumes a 60-minute lifetime and renews them shortly before it ends, and one that cannot, such as a FileIO for a scan plan's credentials, treats them as never expiring. A token whose actual lifetime is shorter than that fails with authorization errors until it is renewed, so catalogs that vend adls.token should send expiration-time with it. For the storage account it is vended for, a vended token replaces the SAS token and connection string configured for that account and any shared key, which would otherwise take precedence over it; a shared key is not scoped to an account, so one configured for another account would be used for this account too. The SAS tokens and connection strings of other accounts are kept. Expiries configured for other credentials, such as a SAS expiry or an expiration-time in the catalog or table configuration, do not affect when the token is renewed.

Note on cross-client parity. The two properties in step 5 diverge from the other clients. adls.auth.managed-identity.enabled is Go-specific: neither Iceberg Java nor PyIceberg defines a property that forces ManagedIdentityCredential, so catalog properties that set it do not carry over. adls.client-id collides in name only — PyIceberg reads it as a service principal client ID used together with adls.client-secret and adls.tenant-id, neither of which iceberg-go defines (see the PyIceberg ADLS configuration), while Java's AzureProperties defines no adls.client-id at all. Here it is only ever read as a user-assigned managed identity ID, and only when step 5 is selected.

Tuning properties:

Environment variables

iceberg-go reads only a small set of environment variables directly. AWS / GCP / Azure credentials flow through the respective SDKs, not through iceberg-go-defined env vars.

There is no PYICEBERG_*-style env var convention. Use the YAML config file or pass iceberg.Properties to overrides programmatically.

Concurrency

Pluggability

Two registries are user-extensible. The third (LocationProvider) is currently informational.

IO scheme registry

Register a custom URL scheme from an init function with io.MustRegister:

import (
    "context"
    "net/url"

    "github.com/apache/iceberg-go/io"
)

func init() {
    io.MustRegister("myfs", func(ctx context.Context, parsed *url.URL, props map[string]string) (io.IO, error) {
        return newMyFS(parsed, props)
    })
}

io.Register and io.MustRegister both panic for invalid or duplicate global schemes, preserving the existing registration contract. registry.Register on an independent registry returns ErrInvalidSchemeFactory or ErrSchemeAlreadyRegistered instead. Built-in schemes: file, "" (the empty scheme). Cloud schemes (s3, gs, abfs, etc.) are registered only when the matching backend package under io/gocloud is blank-imported.

io.GetRegisteredSchemes() returns the current scheme list; io.Unregister(scheme) removes one.

To register schemes at runtime without changing other users of the same process, create an independent registry (catalogs and view.CreateView still resolve through the default registry):

registry := io.NewRegistry()
if err := registry.Register("myfs", myFactory); err != nil {
    return err
}
fs, err := registry.LoadFS(ctx, props, "myfs://bucket/key")

io.NewRegistry() copies schemes already registered globally (including blank-imported cloud backends) but keeps later registrations independent. registry.Clone() makes an isolated copy of another registry. registry.Names() lists the schemes registered in that instance. registry.LoadFSFunc() is the lazy factory equivalent. Global io.LoadFS() and io.LoadFSFunc() continue using the default registry.

Catalog type registry

Register a custom catalog type with catalog.Register:

import (
    "context"

    "github.com/apache/iceberg-go"
    "github.com/apache/iceberg-go/catalog"
)

func init() {
    catalog.Register("mycatalog", catalog.RegistrarFunc(
        func(ctx context.Context, name string, props iceberg.Properties) (catalog.Catalog, error) {
            return newMyCatalog(name, props)
        },
    ))
}

After registration, catalog.Load(ctx, "default", iceberg.Properties{"type": "mycatalog", ...}) will route to the factory. Custom catalogs must implement Close() error to satisfy catalog.Catalog, and callers must close them when finished. Built-in types: rest, hive, glue, sql, hadoop.

catalog.GetRegisteredCatalogs() returns the current list; catalog.Unregister(catalogType) removes one.

LocationProvider

table/locations.go defines the LocationProvider interface and ships two implementations: simpleLocationProvider (default) and objectStoreLocationProvider (selected by write.object-storage.enabled = true). The provider is chosen by table properties and is not user-pluggable today.

Table write properties

Property key constants are in table/properties.go and Parquet keys in table/internal/parquet_files.go. The keys below have verified read sites in non-test code.

Format and file sizing

Scan reads

The catalog's FileIO properties configure manifest caching for all built-in catalogs. With direct table.New calls, use WithScanPlanningIOProperties or WithSavedConfig; saved config wins when the same key is present in both. Matching expiration interval, total byte limit and per-manifest byte limit preserve warm cache entries across refresh and commit. Changing a limit replaces the cache.

Unlike Java, which rejects invalid manifest-cache limits, Go warns and disables the content cache for negative or unparsable expiration or non-positive/unparsable size limits.

The manifest-list cache retains at most 64 snapshots and 32,768 manifest descriptors in total per table handle. These are count limits, not byte limits: memory use also depends on manifest paths, partition summaries, and key metadata. Refreshing the table replaces its manifest-list cache.

The optional manifest-content cache is separate. It stores immutable raw manifest bytes and still decodes, projects, and filters entries independently for each scan. Cache misses for the same manifest share one in-flight read. If population fails, its producer and each existing waiter independently fall back to one ordinary FileIO open; if the producer was canceled, live waiters can start another shared load. Failed content is not retained, so a later scan can retry. Manifests larger than the configured per-file limit bypass the cache. The current integration covers local scan planning; manifest lists continue to use the decoded manifest-list cache.

The Java-compatible read.split.planning-lookback and read.split.open-file-cost names and defaults are also exported by the Go API for shared configuration, but are reserved for future task-group planning and are not currently read.

Metrics and metadata lifecycle

Manifest and commit

Snapshot retention

The unprefixed retention property names are accepted as compatibility fallbacks. The history.expire.* properties take precedence when both forms are set.

Delete mode

Object-store data layout

Parquet writer

All defined in table/internal/parquet_files.go and read by the Parquet writer:

The bloom filter and dictionary toggles are parsed the way Iceberg Java parses them: only true (case-insensitive) enables, so any other value — including 1 — reads as false.

Parquet reader