Configuration

This page documents how to configure Apache Iceberg Go: the CLI's YAML config file, per-catalog Option surfaces, file-system credentials, table write properties, concurrency, and how to plug in custom catalogs and IO backends.

Only properties and options that the iceberg-go code actually reads are listed. Properties defined in the Apache Iceberg spec but not yet wired into iceberg-go are intentionally omitted - check pkg.go.dev/github.com/apache/iceberg-go for the latest read sites.

CLI configuration file

The iceberg CLI loads catalog defaults from ~/.iceberg-go.yaml (override the directory with GOICEBERG_HOME). The schema, defined in config/config.go, is:

default-catalog: default
max-workers: 5
catalog:
  default:
    type: rest
    uri: https://example.com/iceberg
    warehouse: s3://my-bucket/warehouse
    credential: <client-id>:<client-secret>
    output: text
    rest:
      sigv4-enabled: false
      signing-name: ""
      signing-region: ""

Catalog options

Each catalog package exposes its own functional Option set. The lists below reflect the public option surface; pkg.go.dev is authoritative for the current set.

REST (catalog/rest)

The most option-rich surface. Source: catalog/rest/options.go.

Hive (catalog/hive)

Source: catalog/hive/options.go.

• WithURI(uri string) - Thrift URI for the Hive Metastore (e.g. thrift://127.0.0.1:9083).
• WithWarehouse(warehouse string)
• WithProperties(props iceberg.Properties)

Glue (catalog/glue)

Source: catalog/glue/options.go.

• WithAwsConfig(cfg aws.Config) - AWS SDK v2 config; respects the AWS default credential chain.
• WithAwsProperties(props AwsProperties) - explicit overrides for region/endpoint/access keys.

SQL (catalog/sql)

The SQL catalog has no functional-option surface. Construct it with NewCatalog:

db, _ := sql.Open(sqliteshim.ShimName, "file:catalog.db")
cat, err := sqlcat.NewCatalog("default", db, sqlcat.SQLite, iceberg.Properties{
    "warehouse": "file:///tmp/warehouse",
})

Supported dialects: sqlcat.Postgres, sqlcat.MySQL, sqlcat.SQLite, sqlcat.MSSQL, sqlcat.Oracle (catalog/sql/sql.go:50).

Shared options on the base catalog package

Operations that create or update tables/views accept these (catalog/catalog.go):

• WithLocation, WithPartitionSpec, WithSortOrder, WithProperties, WithStagedUpdates
• View-specific: WithViewLocation, WithViewProperties

File-system credentials

iceberg-go registers the local file system (file://) automatically. Cloud schemes are not registered until you add a blank import:

import _ "github.com/apache/iceberg-go/io/gocloud"

The init() function in io/gocloud/register.go registers s3, s3a, s3n, oss, gs, abfs, abfss, wasb, and wasbs. Without the blank import, these schemes return ErrIOSchemeNotFound with a hint to add the import.

All credential and tuning property keys are constants in io/config.go. They can be supplied through table properties, catalog properties, or per-call iceberg.Properties arguments depending on context.

S3

Authentication is resolved in this order (io/gocloud/s3.go):

1. Static credentials in properties: s3.access-key-id + s3.secret-access-key (+ optional s3.session-token).
2. The standard AWS SDK v2 default credential chain - environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN), ~/.aws/credentials, container/IAM role.

Tuning properties:

Google Cloud Storage

Authentication resolution (io/gocloud/gcs.go):

1. Explicit JSON key bytes via gcs.jsonkey or path via gcs.keypath.
2. Optional gcs.credtype selecting one of service_account, authorized_user, impersonated_service_account, external_account.
3. The GCP default credentials chain (gcp.DefaultCredentials) - falls back to anonymous if no creds are found.

Tuning properties:

Azure Data Lake Storage / Blob

Authentication is selected based on the property keys present (io/gocloud/azure.go):

1. Shared key: both adls.auth.shared-key.account.name and adls.auth.shared-key.account.key set.
2. Per-host SAS token: adls.sas-token.<hostname> (prefix-matched against the storage account host).
3. Per-host connection string: adls.connection-string.<hostname>.
4. Managed identity: adls.auth.managed-identity.enabled set to a truthy value.

Tuning properties:

Environment variables

iceberg-go reads only a small set of environment variables directly. AWS / GCP / Azure credentials flow through the respective SDKs, not through iceberg-go-defined env vars.

There is no PYICEBERG_*-style env var convention. Use the YAML config file or pass iceberg.Properties to overrides programmatically.

Concurrency

Pluggability

Two registries are user-extensible. The third (LocationProvider) is currently informational.

IO scheme registry

Register a custom URL scheme with io.Register:

import (
    "context"
    "net/url"

    "github.com/apache/iceberg-go/io"
)

func init() {
    io.Register("myfs", func(ctx context.Context, parsed *url.URL, props map[string]string) (io.IO, error) {
        return newMyFS(parsed, props)
    })
}

io.Register panics on nil factory or duplicate scheme. Built-in schemes: file, "" (the empty scheme). Cloud schemes (s3, gs, abfs, etc.) are registered by io/gocloud only when its package is blank-imported.

io.GetRegisteredSchemes() returns the current scheme list; io.Unregister(scheme) removes one.

Catalog type registry

Register a custom catalog type with catalog.Register:

import (
    "context"

    "github.com/apache/iceberg-go"
    "github.com/apache/iceberg-go/catalog"
)

func init() {
    catalog.Register("mycatalog", catalog.RegistrarFunc(
        func(ctx context.Context, name string, props iceberg.Properties) (catalog.Catalog, error) {
            return newMyCatalog(name, props)
        },
    ))
}

After registration, catalog.Load(ctx, "default", iceberg.Properties{"type": "mycatalog", ...}) will route to the factory. Built-in types: rest, hive, glue, sql, hadoop.

catalog.GetRegisteredCatalogs() returns the current list; catalog.Unregister(catalogType) removes one.

LocationProvider

table/locations.go defines the LocationProvider interface and ships two implementations: simpleLocationProvider (default) and objectStoreLocationProvider (selected by write.object-storage.enabled = true). The provider is chosen by table properties and is not user-pluggable today.

Table write properties

Property key constants are in table/properties.go and Parquet keys in table/internal/parquet_files.go. The keys below have verified read sites in non-test code.

Format and file sizing

Metrics and metadata lifecycle

Manifest and commit

Snapshot retention

Delete mode

Object-store data layout

Parquet writer

All defined in table/internal/parquet_files.go and read by the Parquet writer:

Parquet reader