Iceberg Go

Apache Iceberg Go is a Go-native implementation of Apache Iceberg, the open table format for analytic datasets. Read and write Iceberg tables from Go services and tooling without a JVM.

Where to start

• Install - get the library and CLI on your machine.
• CLI - inspect tables, run maintenance, manage refs.
• API - construct catalogs, scan tables, write Arrow data, evolve schemas.
• Configuration - YAML config file, catalog options, FileIO and credentials, table properties.

What works today

A capability matrix - filesystem, metadata operations, catalog support, and write operations - lives on the dedicated Feature Status page.

Beyond Go

Apache Iceberg is multi-language. PyIceberg, iceberg-rust, iceberg-cpp, and the Java reference implementation all target the same spec - see Other Iceberg implementations for cross-links.

Help and contribution

• Questions and discussion: Community
• Contributing code or docs: Contributing
• Cutting or verifying a release: Releases

The canonical Iceberg specification, terminology, and multi-engine policy live with the main project at iceberg.apache.org. This site covers what is specific to the Go implementation.

Install

In this quickstart, we’ll glean insights from code segments and learn how to:

Requirements

Installation

To install iceberg-go package, you need to install Go and set your Go workspace first. If you don't have a go.mod file, create it with go mod init gin.

1. Download and install it:

go get -u github.com/apache/iceberg-go

2. Import it in your code:

import "github.com/apache/iceberg-go"

Getting Started

This walkthrough takes you from go get to a fully written-and-read Iceberg table in a few minutes, using a local SQLite-backed catalog so you do not need any external services. By the end you will have:

• An Iceberg catalog stored in a SQLite file
• A table with an Arrow schema
• Data written from Apache Arrow Go and read back
• A schema evolution committed
• A branch created on top of the table

If you would rather see all the operations as a reference, jump straight to the API.

1. Install

go mod init iceberg-go-tutorial
go get github.com/apache/iceberg-go@latest
go get github.com/apache/arrow-go/v18@latest
go get github.com/uptrace/bun/driver/sqliteshim@latest

iceberg-go itself only registers the local file system. For S3, GCS, or Azure Blob you would also blank-import the matching backend package, such as github.com/apache/iceberg-go/io/gocloud/s3. We are staying on local disk for this tutorial.

2. Open a local catalog

The SQL catalog stores its metadata in a database, and the actual data files live under a "warehouse" directory. We will use SQLite for both the catalog DB and a local folder for the warehouse.

package main

import (
    "context"
    "database/sql"
    "log"

    "github.com/apache/iceberg-go"
    sqlcat "github.com/apache/iceberg-go/catalog/sql"
    "github.com/uptrace/bun/driver/sqliteshim"
)

func openCatalog(ctx context.Context) (*sqlcat.Catalog, error) {
    db, err := sql.Open(sqliteshim.ShimName, "file:./tutorial-catalog.db?cache=shared")
    if err != nil {
        return nil, err
    }
    return sqlcat.NewCatalog("default", db, sqlcat.SQLite, iceberg.Properties{
        "warehouse":            "file:///tmp/iceberg-tutorial",
        "init_catalog_tables":  "true",
    })
}

init_catalog_tables: "true" lets the catalog create its bookkeeping tables (iceberg_tables, iceberg_namespace_properties) on first use. Make sure /tmp/iceberg-tutorial is writable.

3. Create a namespace and a table

Iceberg organizes tables under namespaces (the analog of databases). We will create one and put a table inside it.

import (
    "github.com/apache/iceberg-go/catalog"
    "github.com/apache/iceberg-go/table"
)

func createTrips(ctx context.Context, cat *sqlcat.Catalog) (*table.Table, error) {
    ns := table.Identifier{"taxi"}
    if err := cat.CreateNamespace(ctx, ns, nil); err != nil {
        return nil, err
    }

    schema := iceberg.NewSchema(1,
        iceberg.NestedField{ID: 1, Name: "trip_id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
        iceberg.NestedField{ID: 2, Name: "fare", Type: iceberg.PrimitiveTypes.Float64, Required: false},
        iceberg.NestedField{ID: 3, Name: "borough", Type: iceberg.PrimitiveTypes.String, Required: false},
    )

    ident := catalog.ToIdentifier("taxi", "trips")
    return cat.CreateTable(ctx, ident, schema)
}

If the namespace already exists, CreateNamespace returns an error - the example skips error handling for brevity; production code should check for iceberg.ErrAlreadyExists and recover.

4. Write some Arrow data

The write path takes either a streaming array.RecordReader or a fully materialized arrow.Table. We will build a small Arrow table inline and append it.

import (
    "github.com/apache/arrow-go/v18/arrow"
    "github.com/apache/arrow-go/v18/arrow/array"
    "github.com/apache/arrow-go/v18/arrow/memory"
)

func writeSomeTrips(ctx context.Context, tbl *table.Table) (*table.Table, error) {
    mem := memory.NewGoAllocator()

    arrowSchema := arrow.NewSchema([]arrow.Field{
        {Name: "trip_id", Type: arrow.PrimitiveTypes.Int64, Nullable: false},
        {Name: "fare", Type: arrow.PrimitiveTypes.Float64, Nullable: true},
        {Name: "borough", Type: arrow.BinaryTypes.String, Nullable: true},
    }, nil)

    b := array.NewRecordBuilder(mem, arrowSchema)
    defer b.Release()

    b.Field(0).(*array.Int64Builder).AppendValues([]int64{1, 2, 3}, nil)
    b.Field(1).(*array.Float64Builder).AppendValues([]float64{12.50, 8.75, 22.10}, nil)
    b.Field(2).(*array.StringBuilder).AppendValues([]string{"Manhattan", "Brooklyn", "Queens"}, nil)

    rec := b.NewRecord()
    defer rec.Release()

    arrowTbl := array.NewTableFromRecords(arrowSchema, []arrow.Record{rec})
    defer arrowTbl.Release()

    return tbl.AppendTable(ctx, arrowTbl, 1024 /* batchSize */, nil)
}

AppendTable returns a refreshed *table.Table reflecting the new snapshot.

5. Read the data back

func readAll(ctx context.Context, tbl *table.Table) error {
    arrowTbl, err := tbl.Scan().ToArrowTable(ctx)
    if err != nil {
        return err
    }
    defer arrowTbl.Release()

    log.Printf("read %d rows in %d columns", arrowTbl.NumRows(), arrowTbl.NumCols())
    return nil
}

For larger tables, use streaming so only one batch is in memory at a time:

arrowSchema, batches, err := tbl.Scan().ToArrowRecords(ctx)
if err != nil {
    return err
}
log.Printf("scan schema: %s", arrowSchema)

for batch, err := range batches {
    if err != nil {
        return err
    }
    log.Printf("batch with %d rows", batch.NumRows())
    batch.Release()
}

6. Filter and project

Use the predicate DSL to push filters down into the scan, and WithSelectedFields to project columns:

filter := iceberg.GreaterThan(iceberg.Reference("fare"), float64(10.0))

arrowTbl, err := tbl.Scan(
    table.WithSelectedFields("trip_id", "fare"),
    table.WithRowFilter(filter),
).ToArrowTable(ctx)

7. Evolve the schema

Most schema changes go through a transaction. Add a tip column:

import "github.com/apache/iceberg-go/table"

func addTipColumn(ctx context.Context, tbl *table.Table) (*table.Table, error) {
    txn := tbl.NewTransaction()

    err := table.NewUpdateSchema(txn, true /* caseSensitive */, false /* allowIncompatible */).
        AddColumn([]string{"tip"}, iceberg.PrimitiveTypes.Float64, "Tip in dollars", false, nil).
        Commit()
    if err != nil {
        return nil, err
    }

    return txn.Commit(ctx)
}

The returned table has the new schema and a fresh metadata file.

8. Branch the table

A branch is a named ref that points at a snapshot. Creating one requires a metadata commit that adds the ref - NewTransactionOnBranch only writes to a branch that already exists. Two steps:

import "github.com/apache/iceberg-go/table"

// Step 1: create the branch via Catalog.CommitTable.
func createExperimentBranch(ctx context.Context, cat *sqlcat.Catalog, tbl *table.Table) (*table.Table, error) {
    snap := tbl.CurrentSnapshot()
    update := table.NewSetSnapshotRefUpdate(
        "experiment", snap.SnapshotID, table.BranchRef,
        0 /* maxRefAgeMs */, 0 /* maxSnapshotAgeMs */, 0 /* minSnapshotsToKeep */,
    )
    reqs := []table.Requirement{
        table.AssertTableUUID(tbl.Metadata().TableUUID()),
        table.AssertRefSnapshotID("experiment", nil), // branch must not yet exist
    }
    if _, _, err := cat.CommitTable(ctx, tbl.Identifier(), reqs, []table.Update{update}); err != nil {
        return nil, err
    }
    // Reload so subsequent transactions see the new ref.
    return cat.LoadTable(ctx, tbl.Identifier())
}

// Step 2: open a transaction on the branch and write.
func writeOnBranch(ctx context.Context, tbl *table.Table, arrowTbl arrow.Table) (*table.Table, error) {
    txn, err := tbl.NewTransactionOnBranch("experiment")
    if err != nil {
        return nil, err
    }
    if err := txn.AppendTable(ctx, arrowTbl, 1024, nil); err != nil {
        return nil, err
    }
    return txn.Commit(ctx)
}

Reads can target the branch with tbl.Scan().UseRef("experiment").

Where to go next

• API - the full Go surface for catalogs, tables, scans, writes, transactions, schema and partition evolution, snapshot management, maintenance, and views.
• Configuration - cloud credentials, table properties, concurrency, custom catalog and IO registration.
• CLI - inspect, expire, compact, branch, and tag from the command line.
• Row Filter Syntax and Expression DSL - the predicate DSL in detail.

The full Iceberg specification, terminology, and engine integration policy live with the main project at iceberg.apache.org.

Configuration

This page documents how to configure Apache Iceberg Go: the CLI's YAML config file, per-catalog Option surfaces, file-system credentials, table write properties, concurrency, and how to plug in custom catalogs and IO backends.

Only properties and options that the iceberg-go code actually reads are listed. Properties defined in the Apache Iceberg spec but not yet wired into iceberg-go are intentionally omitted - check pkg.go.dev/github.com/apache/iceberg-go for the latest read sites.

CLI configuration file

The iceberg CLI loads catalog defaults from ~/.iceberg-go.yaml (override the directory with GOICEBERG_HOME). The schema, defined in config/config.go, is:

default-catalog: default
max-workers: 5
catalog:
  default:
    type: rest
    uri: https://example.com/iceberg
    warehouse: s3://my-bucket/warehouse
    credential: <client-id>:<client-secret>
    output: text
    aws-profile: ""
    rest:
      sigv4-enabled: false
      signing-name: ""
      signing-region: ""

Catalog options

Each catalog package exposes its own functional Option set. The lists below reflect the public option surface; pkg.go.dev is authoritative for the current set.

REST (catalog/rest)

The most option-rich surface. Source: catalog/rest/options.go.

AWS SigV4 is an optional backend. catalog/rest no longer links the AWS SDK. To sign REST requests with SigV4, pull in the catalog/rest/sigv4 sub-package one of two ways:

• Property or ambient credentials: add a blank import _ "github.com/apache/iceberg-go/catalog/rest/sigv4" and enable signing with WithSigV4 / WithSigV4RegionSvc or the rest.sigv4-enabled property. The backend signs with the s3.* credential properties when set (falling back to the rest.* aliases), otherwise with the AWS default credential chain.
• Explicit config: pass sigv4.WithAWSConfig(cfg) to NewCatalog alongside WithSigV4 / WithSigV4RegionSvc (no blank import needed). It supplies the aws.Config; the signing service comes from those options (or a server /v1/config override), and so does the region when one is set. Otherwise the region falls back to cfg.Region.

The former rest.WithAwsConfig(aws.Config) has been removed in favor of these paths. Because signing is enabled by WithSigV4 / WithSigV4RegionSvc (or the rest.sigv4-enabled property) rather than by supplying a config, a program that loads a REST catalog with rest.sigv4-enabled=true now fails at startup until the catalog/rest/sigv4 backend is blank-imported; the error names the import to add.

Custom token sources (Google BigLake)

WithCredential uses the OAuth2 client-credentials flow against the catalog's token endpoint. For other identity providers, wrap any oauth2.TokenSource with NewOauth2AuthManager and pass it to WithAuthManager. Google's BigLake Iceberg REST catalog, for example, takes Application Default Credentials:

ts, err := google.DefaultTokenSource(ctx, "https://www.googleapis.com/auth/cloud-platform")
if err != nil {
	return err
}
cat, err := rest.NewCatalog(ctx, "biglake", "https://biglake.googleapis.com/iceberg/v1/restcatalog",
	rest.WithAuthManager(rest.NewOauth2AuthManager(ts)),
	rest.WithWarehouseLocation("gs://lakehouse-public-data"),
	rest.WithHeaders(map[string]string{"x-goog-user-project": "my-project"}))

The manager reuses a token until it expires, then asks the source for a new one. That refresh runs on the HTTP client the source was built with. For google.DefaultTokenSource that is the client in the context's oauth2.HTTPClient value, or http.DefaultClient, which has no timeout. Unlike Java's rest.auth.type=google, this is configured in code; iceberg-go has no rest.auth.type property.

Metrics reporting

The REST catalog can POST scan and commit metrics to the catalog's .../tables/{table}/metrics endpoint. These properties are passed as client-supplied catalog properties.

Note on cross-client parity. Iceberg Java defaults metrics reporting to on (METRICS_REPORTING_ENABLED_DEFAULT=true); iceberg-go deliberately defaults it off, so no client emits new outbound telemetry unless it explicitly opts in. A Java client that never set the flag will therefore stop reporting when ported here until rest-metrics-reporting-enabled=true is set. Enablement is read from client properties only — a server cannot turn on reporting the client never asked for — but an explicit server override setting the flag to false is honored (disabling is always safe), so an operator can suppress reporting fleet-wide. rest-metrics-reporting-timeout-ms is a Go-specific extension with no Java, PyIceberg, or Rust equivalent; do not assume parity on it in multi-client deployments.

Hive (catalog/hive)

Source: catalog/hive/options.go.

• WithURI(uri string) - Thrift URI for the Hive Metastore (e.g. thrift://127.0.0.1:9083).
• WithWarehouse(warehouse string)
• WithProperties(props iceberg.Properties)

Lock-check catalog properties (passed via WithProperties / catalog.Load props):

Invalid values (non-positive waits/retries, or min ≥ max) are ignored and the previous/default settings are kept.

Glue (catalog/glue)

Source: catalog/glue/options.go.

• WithAWSConfig(cfg aws.Config) - AWS SDK v2 config; respects the AWS default credential chain.
• WithAWSProperties(props AWSProperties) - catalog properties, such as glue.id and the s3.* FileIO keys. NewCatalog ignores the glue.region, endpoint, credential, and retry keys; only catalog.Load reads them.

SQL (catalog/sql)

The SQL catalog has no functional-option surface. Construct it with NewCatalog:

db, _ := sql.Open(sqliteshim.ShimName, "file:catalog.db")
cat, err := sqlcat.NewCatalog("default", db, sqlcat.SQLite, iceberg.Properties{
    "warehouse": "file:///tmp/warehouse",
})

Supported dialects: sqlcat.Postgres, sqlcat.MySQL, sqlcat.SQLite, sqlcat.MSSQL, sqlcat.Oracle (catalog/sql/sql.go:50).

Shared options on the base catalog package

Operations that create or update tables/views accept these (catalog/catalog.go):

• WithLocation, WithPartitionSpec, WithSortOrder, WithProperties, WithStagedUpdates
• View-specific: WithViewLocation, WithViewProperties

File-system credentials

iceberg-go registers the local file system (file://) automatically. Cloud schemes are not registered until you add a blank import:

import _ "github.com/apache/iceberg-go/io/gocloud/s3"    // s3, s3a, s3n, oss
import _ "github.com/apache/iceberg-go/io/gocloud/gcs"   // gs
import _ "github.com/apache/iceberg-go/io/gocloud/azure" // abfs, abfss, wasb, wasbs

Each backend package links its own cloud SDK. Importing io/gocloud registers all three at once and links all cloud SDKs.

Without a matching blank import, these schemes return ErrIOSchemeNotFound with a hint naming the package to import.

All credential and tuning property keys are constants in io/config.go. They can be supplied through table properties, catalog properties, or per-call iceberg.Properties arguments depending on context.

S3

Authentication is resolved in this order (io/gocloud/s3/s3.go):

1. Static credentials in properties: s3.access-key-id + s3.secret-access-key (+ optional s3.session-token).
2. The standard AWS SDK v2 default credential chain - environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN), ~/.aws/credentials, container/IAM role.

Tuning properties:

Google Cloud Storage

Authentication resolution (io/gocloud/gcs/gcs.go):

1. Explicit JSON key bytes via gcs.jsonkey or path via gcs.keypath.
2. Optional gcs.credtype selecting one of service_account, authorized_user, impersonated_service_account, external_account.
3. The GCP default credentials chain (gcp.DefaultCredentials) - falls back to anonymous if no creds are found.

Tuning properties:

Azure Data Lake Storage / Blob

Authentication is selected in the following order (io/gocloud/azure/azure.go); the first applicable option wins, and if none of 1-5 apply the default credential chain (6) is used:

1. Shared key: a nonempty adls.auth.shared-key.account.name selects this path and requires a nonempty adls.auth.shared-key.account.key.
2. Per-host SAS token: adls.sas-token.<hostname>, where <hostname> exactly matches the storage account hostname (for example, myaccount.dfs.core.windows.net).
3. Per-account connection string: adls.connection-string.<account-name> (for example, adls.connection-string.myaccount).
4. Static access token: a nonempty adls.token, supplied without the Bearer prefix.
5. Managed identity: adls.auth.managed-identity.enabled set to exactly "true" selects this path and uses ManagedIdentityCredential directly. Set adls.client-id to select a user-assigned managed identity; otherwise, the system-assigned managed identity is used.
6. Default credential chain: DefaultAzureCredential, which includes managed identity among its credential sources.

adls.token accepts an OAuth2 access token for Azure Storage, including a token supplied by a catalog. An empty value is ignored. A rejected or expired token does not fall back to managed identity or the default credential chain.

The token is static and is not refreshed. As in Java and PyIceberg, the credential reports a one-hour cache window to the Azure SDK each time the SDK requests the token. This does not extend the token's actual validity: Azure enforces its expiration. Callers must obtain a new token and recreate the FileIO when it expires; changing the original properties map does not update an existing FileIO.

A REST catalog can also vend adls.token as a storage credential. The token carries no expiry of its own, so a vended token's expiry comes from expiration-time in the same credential when the catalog supplies one. Without it, a table FileIO that can fetch new credentials assumes a 60-minute lifetime and renews them shortly before it ends, and one that cannot, such as a FileIO for a scan plan's credentials, treats them as never expiring. A token whose actual lifetime is shorter than that fails with authorization errors until it is renewed, so catalogs that vend adls.token should send expiration-time with it. For the storage account it is vended for, a vended token replaces the SAS token and connection string configured for that account and any shared key, which would otherwise take precedence over it; a shared key is not scoped to an account, so one configured for another account would be used for this account too. The SAS tokens and connection strings of other accounts are kept. Expiries configured for other credentials, such as a SAS expiry or an expiration-time in the catalog or table configuration, do not affect when the token is renewed.

Note on cross-client parity. The two properties in step 5 diverge from the other clients. adls.auth.managed-identity.enabled is Go-specific: neither Iceberg Java nor PyIceberg defines a property that forces ManagedIdentityCredential, so catalog properties that set it do not carry over. adls.client-id collides in name only — PyIceberg reads it as a service principal client ID used together with adls.client-secret and adls.tenant-id, neither of which iceberg-go defines (see the PyIceberg ADLS configuration), while Java's AzureProperties defines no adls.client-id at all. Here it is only ever read as a user-assigned managed identity ID, and only when step 5 is selected.

Tuning properties:

Environment variables

iceberg-go reads only a small set of environment variables directly. AWS / GCP / Azure credentials flow through the respective SDKs, not through iceberg-go-defined env vars.

There is no PYICEBERG_*-style env var convention. Use the YAML config file or pass iceberg.Properties to overrides programmatically.

Concurrency

Pluggability

Two registries are user-extensible. The third (LocationProvider) is currently informational.

IO scheme registry

Register a custom URL scheme from an init function with io.MustRegister:

import (
    "context"
    "net/url"

    "github.com/apache/iceberg-go/io"
)

func init() {
    io.MustRegister("myfs", func(ctx context.Context, parsed *url.URL, props map[string]string) (io.IO, error) {
        return newMyFS(parsed, props)
    })
}

io.Register and io.MustRegister both panic for invalid or duplicate global schemes, preserving the existing registration contract. registry.Register on an independent registry returns ErrInvalidSchemeFactory or ErrSchemeAlreadyRegistered instead. Built-in schemes: file, "" (the empty scheme). Cloud schemes (s3, gs, abfs, etc.) are registered only when the matching backend package under io/gocloud is blank-imported.

io.GetRegisteredSchemes() returns the current scheme list; io.Unregister(scheme) removes one.

To register schemes at runtime without changing other users of the same process, create an independent registry (catalogs and view.CreateView still resolve through the default registry):

registry := io.NewRegistry()
if err := registry.Register("myfs", myFactory); err != nil {
    return err
}
fs, err := registry.LoadFS(ctx, props, "myfs://bucket/key")

io.NewRegistry() copies schemes already registered globally (including blank-imported cloud backends) but keeps later registrations independent. registry.Clone() makes an isolated copy of another registry. registry.Names() lists the schemes registered in that instance. registry.LoadFSFunc() is the lazy factory equivalent. Global io.LoadFS() and io.LoadFSFunc() continue using the default registry.

Catalog type registry

Register a custom catalog type with catalog.Register:

import (
    "context"

    "github.com/apache/iceberg-go"
    "github.com/apache/iceberg-go/catalog"
)

func init() {
    catalog.Register("mycatalog", catalog.RegistrarFunc(
        func(ctx context.Context, name string, props iceberg.Properties) (catalog.Catalog, error) {
            return newMyCatalog(name, props)
        },
    ))
}

After registration, catalog.Load(ctx, "default", iceberg.Properties{"type": "mycatalog", ...}) will route to the factory. Custom catalogs must implement Close() error to satisfy catalog.Catalog, and callers must close them when finished. Built-in types: rest, hive, glue, sql, hadoop.

catalog.GetRegisteredCatalogs() returns the current list; catalog.Unregister(catalogType) removes one.

LocationProvider

table/locations.go defines the LocationProvider interface and ships two implementations: simpleLocationProvider (default) and objectStoreLocationProvider (selected by write.object-storage.enabled = true). The provider is chosen by table properties and is not user-pluggable today.

Table write properties

Property key constants are in table/properties.go and Parquet keys in table/internal/parquet_files.go. The keys below have verified read sites in non-test code.

Format and file sizing

Scan reads

The catalog's FileIO properties configure manifest caching for all built-in catalogs. With direct table.New calls, use WithScanPlanningIOProperties or WithSavedConfig; saved config wins when the same key is present in both. Matching expiration interval, total byte limit and per-manifest byte limit preserve warm cache entries across refresh and commit. Changing a limit replaces the cache.

Unlike Java, which rejects invalid manifest-cache limits, Go warns and disables the content cache for negative or unparsable expiration or non-positive/unparsable size limits.

The manifest-list cache retains at most 64 snapshots and 32,768 manifest descriptors in total per table handle. These are count limits, not byte limits: memory use also depends on manifest paths, partition summaries, and key metadata. Refreshing the table replaces its manifest-list cache.

The optional manifest-content cache is separate. It stores immutable raw manifest bytes and still decodes, projects, and filters entries independently for each scan. Cache misses for the same manifest share one in-flight read. If population fails, its producer and each existing waiter independently fall back to one ordinary FileIO open; if the producer was canceled, live waiters can start another shared load. Failed content is not retained, so a later scan can retry. Manifests larger than the configured per-file limit bypass the cache. The current integration covers local scan planning; manifest lists continue to use the decoded manifest-list cache.

The Java-compatible read.split.planning-lookback and read.split.open-file-cost names and defaults are also exported by the Go API for shared configuration, but are reserved for future task-group planning and are not currently read.

Metrics and metadata lifecycle

Manifest and commit

Snapshot retention

The unprefixed retention property names are accepted as compatibility fallbacks. The history.expire.* properties take precedence when both forms are set.

Delete mode

Object-store data layout

Parquet writer

All defined in table/internal/parquet_files.go and read by the Parquet writer:

The bloom filter and dictionary toggles are parsed the way Iceberg Java parses them: only true (case-insensitive) enables, so any other value — including 1 — reads as false.

Parquet reader

CLI

Run go build ./cmd/iceberg from the root of this repository to build the CLI executable, alternately you can run go install github.com/apache/iceberg-go/cmd/iceberg to install it to the bin directory of your GOPATH.

The iceberg CLI usage is very similar to pyiceberg CLI.
You can pass the catalog URI with the --uri argument.

Connecting to a catalog

Start a local REST catalog (the default --catalog type):

docker pull apache/iceberg-rest-fixture:latest
docker run -p 8181:8181 apache/iceberg-rest-fixture:latest

and run the iceberg CLI pointing to the REST API server:

 ./iceberg --uri http://0.0.0.0:8181 list
┌─────┐
| IDs |
| --- |
└─────┘

Catalog connection flags are global and apply to every subcommand:

To avoid passing flags every time, define a config file at ~/.iceberg-go.yaml:

default-catalog: default
catalog:
  default:
    type: rest
    uri: http://localhost:8181
    warehouse: s3://my-warehouse

SQL catalog example:

catalog:
  local:
    type: sql
    uri: file:iceberg-catalog.db
    warehouse: file:///tmp/warehouse
    sql-driver: sqliteshim
    sql-dialect: sqlite

Flags on the command line override values from the config file.

Output format

All commands accept --output text (default, human-readable) or --output json (machine-readable, suitable for piping into jq or scripts).

Catalog and namespace commands

Create namespace

./iceberg --uri http://0.0.0.0:8181 create namespace taxitrips

List namespaces

 ./iceberg --uri http://0.0.0.0:8181 list
┌───────────┐
| IDs       |
| --------- |
| taxitrips |
└───────────┘

Create table

Note: only the identity transform is supported for --partition-spec at this moment.

# Create a simple table with REST catalog and object store
./iceberg create table default.table-1 \
        --properties write.format.default=parquet \
        --partition-spec foo \
        --sort-order foo:desc:nulls-last \
        --schema '[{"id":1,"name":"foo","type":"string","required":false},{"id":2,"name":"bar","type":"int","required":true}]' \
        --catalog rest \
        --uri http://localhost:8181
Table default.table-1 created successfully

# Describe the newly created table
./iceberg describe --catalog rest --uri http://localhost:8181 default.table-1
Table format version │ 2
Metadata location    │ s3://warehouse/default/table-1/metadata/00000-f0ccaadd-d988-482e-99da-3a37870288fe.metadata.json
Table UUID           │ 33fa3fac-e638-4335-a085-343c6d9e7de5
Last updated         │ 1753133512562
Sort Order           │ 1: [
                     │ 1 desc nulls-last
                     │ ]
Partition Spec       │ [
                     │  1000: foo: identity(1)
                     │ ]

Current Schema, id=0
├── 1: foo: optional string
└── 2: bar: required int

Current Snapshot │

Snapshots

Properties
key                             │ value
-----------------------------------------
write.format.default            │ parquet
write.parquet.compression-codec │ zstd

Inspecting tables

info — single-screen summary

iceberg info my_db.events

Reports format version, location, current snapshot, schema, partition spec, sort order, snapshot count, ref count, and property count for a table.

snapshots — snapshot history

iceberg snapshots my_db.events

Lists all snapshots with timestamp, parent snapshot, operation (append, overwrite, delete, replace), and added/deleted data file counts.

refs — branches and tags

iceberg refs my_db.events
iceberg refs --type branch my_db.events
iceberg refs --type tag    my_db.events

Lists snapshot refs along with their retention settings (max-ref-age, max-snapshot-age, min-snapshots-to-keep).

partition-stats — partition statistics files

iceberg partition-stats my_db.events                          # current snapshot
iceberg partition-stats --snapshot-id 7234981023498 my_db.events
iceberg partition-stats --all my_db.events                    # all snapshots

files — data files

iceberg files my_db.events                              # current snapshot
iceberg files --history my_db.events                    # all snapshots
iceberg files --scan-planning-mode remote my_db.events  # from a server-side scan plan

By default files reads the manifest list and manifests itself and prints them as a tree: snapshot, manifest, then each manifest entry.

With --scan-planning-mode remote or auto, or when a REST catalog sets scan-planning-mode=server for the table, files plans a scan of the current snapshot instead and lists the data files it returns, each with its delete files. A scan plan carries no manifests, so the tree has no manifest level, and it only shows live files. --history needs every snapshot's manifests and fails in this mode, as does --scan-planning-mode local on a server table.

compact plans its input with a table scan, so it follows the catalog's scan-planning-mode as well. Every other command reads table metadata directly.

schema --show-defaults

iceberg schema --show-defaults my_db.events

Prints the schema and surfaces each field's initial-default and write-default — useful when debugging schema-evolution behavior.

Snapshot maintenance

expire-snapshots

Drop old snapshots so their unreferenced data files become eligible for cleanup.

# Preview
iceberg expire-snapshots --older-than 7d --dry-run my_db.events

# Commit, retaining at least 5 snapshots
iceberg expire-snapshots --older-than 7d --retain-last 5 my_db.events

clean-orphan-files

Remove data files in the table location that are not referenced by any snapshot's manifests (e.g. left behind by failed writes).

iceberg clean-orphan-files --dry-run my_db.events
iceberg clean-orphan-files --older-than 5d my_db.events
iceberg clean-orphan-files --location s3://old-warehouse/my_db/events my_db.events

Rollback

Reset the current snapshot pointer to a previous snapshot. The target must be an ancestor of the current snapshot.

iceberg rollback --snapshot-id 6891234567890 my_db.events

Format upgrade

Upgrade the table format version (metadata-only operation; no data files are rewritten). Refuses downgrades and same-version "upgrades".

iceberg upgrade --dry-run my_db.events 2
iceberg upgrade my_db.events 2

Branches and tags

branch create

iceberg branch create my_db.events ml-experiment-v3

iceberg branch create \
  --snapshot-id 7234981023498 \
  --max-ref-age 30d \
  --max-snapshot-age 7d \
  --min-snapshots-to-keep 10 \
  my_db.events audit-2026-q2

tag create

iceberg tag create my_db.events pre-migration-v4

iceberg tag create \
  --snapshot-id 7234981023498 \
  --max-ref-age 90d \
  my_db.events monthly-backup-may

Automation

Two flags make these commands safe to run from cron jobs or CI:

• --yes skips the interactive prompt. Without it in a non-interactive environment, the CLI exits with stdin is not a terminal: use --yes to confirm in non-interactive mode rather than hanging.
• --output json emits structured output that can be consumed by jq and downstream tooling.

Daily maintenance over every table in a namespace:

#!/bin/bash
TABLES=$(iceberg list my_db --output json | jq -r '.identifiers[].name')
for table in $TABLES; do
  iceberg expire-snapshots --older-than 7d --retain-last 3 --yes \
    --output json "my_db.$table"
  iceberg clean-orphan-files --older-than 3d --yes \
    --output json "my_db.$table"
done

Tag tables before a deploy:

iceberg tag create --yes my_db.events "pre-deploy-$VERSION"
iceberg tag create --yes my_db.users  "pre-deploy-$VERSION"

Audit-only report (no commits):

iceberg expire-snapshots --older-than 7d --dry-run --output json my_db.events \
  | jq '{table, would_expire: .expired_snapshot_count}'

Safety features

Every write command has multiple layers of protection:

• --dry-run — shows the would-be effect without committing. Look for [DRY RUN] in text output or "dry_run": true in JSON.
• --yes — required to skip the prompt; without it, non-interactive shells get an explicit error rather than hanging.
• TTY detection — interactive prompts are only shown when stdout is a terminal.
• Ancestor validation — rollback rejects target snapshots that are not in the current branch's history.
• Version check — upgrade refuses same-version or downgrade requests.

API

The Go API surface for Apache Iceberg Go. New to the project? Walk through the Getting Started tutorial first - the recipes here assume you already have a catalog and a table.

For configuration knobs (catalog options, FileIO credentials, table properties), see Configuration. For predicate construction details, see Row Filter Syntax and Expression DSL.

Catalog

catalog.Catalog is the entry point for everything: namespaces, tables, views.

Constructing a catalog

REST

import (
    "context"

    "github.com/apache/iceberg-go/catalog/rest"
)

cat, err := rest.NewCatalog(context.Background(), "rest", "http://localhost:8181",
    rest.WithOAuthToken("your-token"))

SQL (SQLite, Postgres, MySQL, Oracle, MSSQL)

import (
    "database/sql"

    "github.com/apache/iceberg-go"
    sqlcat "github.com/apache/iceberg-go/catalog/sql"
    "github.com/uptrace/bun/driver/sqliteshim"
)

db, err := sql.Open(sqliteshim.ShimName, "file:catalog.db")
// handle err
cat, err := sqlcat.NewCatalog("default", db, sqlcat.SQLite, iceberg.Properties{
    "warehouse": "file:///tmp/warehouse",
})

Glue

import (
    "github.com/apache/iceberg-go/catalog/glue"
    "github.com/aws/aws-sdk-go-v2/config"
)

awsCfg, err := config.LoadDefaultConfig(context.TODO())
// handle err
cat := glue.NewCatalog(glue.WithAWSConfig(awsCfg))

Hive

import (
    "github.com/apache/iceberg-go"
    "github.com/apache/iceberg-go/catalog/hive"
)

cat, err := hive.NewCatalog(iceberg.Properties{},
    hive.WithURI("thrift://localhost:9083"),
    hive.WithWarehouse("s3://my-bucket/warehouse"))

Hadoop

import (
    "github.com/apache/iceberg-go"
    "github.com/apache/iceberg-go/catalog/hadoop"
    "context"
)

cat, err := hadoop.NewCatalog(context.Background(), "default", "file:///tmp/warehouse", iceberg.Properties{})

Via the registry

catalog.Load looks up the right backend via the type property (or uri scheme as a fallback) plus your ~/.iceberg-go.yaml. Useful when you want runtime selection:

import (
    "github.com/apache/iceberg-go"
    "github.com/apache/iceberg-go/catalog"
)

cat, err := catalog.Load(ctx, "default", iceberg.Properties{
    "type":      "rest",
    "uri":       "http://localhost:8181",
    "warehouse": "s3://my-bucket/warehouse",
})
if err != nil { panic(err) }
defer cat.Close()

Every catalog implements Close() error. Close it when done to release catalog-owned resources. A caller-supplied database connection remains caller-owned.

Namespaces

ns := table.Identifier{"sales"}

err := cat.CreateNamespace(ctx, ns, iceberg.Properties{"owner": "data-team"})

namespaces, err := cat.ListNamespaces(ctx, nil) // []table.Identifier
exists, err := cat.CheckNamespaceExists(ctx, ns)
props, err := cat.LoadNamespaceProperties(ctx, ns)
summary, err := cat.UpdateNamespaceProperties(ctx, ns,
    []string{"deprecated"}, // removals
    iceberg.Properties{"owner": "platform-team"}, // updates
)
err = cat.DropNamespace(ctx, ns)

table.Identifier is []string; use catalog.ToIdentifier("sales", "orders") (or catalog.ToIdentifier("sales.orders")) to build one from string parts.

Tables

Defining a schema

import "github.com/apache/iceberg-go"

schema := iceberg.NewSchema(1,
    iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
    iceberg.NestedField{ID: 2, Name: "name", Type: iceberg.PrimitiveTypes.String, Required: false},
    iceberg.NestedField{ID: 3, Name: "active", Type: iceberg.PrimitiveTypes.Bool, Required: false},
)

For nested types use &iceberg.StructType{...}, &iceberg.ListType{...}, or &iceberg.MapType{...}. Use NewSchemaWithIdentifiers(id, identifierIDs, fields...) to mark identifier columns.

Create

import "github.com/apache/iceberg-go/catalog"

ident := catalog.ToIdentifier("sales", "orders")

tbl, err := cat.CreateTable(ctx, ident, schema,
    catalog.WithLocation("s3://my-bucket/sales/orders"),
    catalog.WithProperties(iceberg.Properties{"owner": "data-team"}),
)

Optional catalog.WithPartitionSpec, catalog.WithSortOrder, and catalog.WithStagedUpdates are also available.

Load, exists, list, drop, rename

tbl, err := cat.LoadTable(ctx, ident)

exists, err := cat.CheckTableExists(ctx, ident)

for ident, err := range cat.ListTables(ctx, table.Identifier{"sales"}) {
    if err != nil { /* ... */ }
    fmt.Println(ident)
}

err = cat.DropTable(ctx, ident)

renamed, err := cat.RenameTable(ctx,
    catalog.ToIdentifier("sales", "orders"),
    catalog.ToIdentifier("sales", "orders_v2"))

ListTables returns an iter.Seq2[table.Identifier, error] that streams results.

Inspecting metadata

tbl.Identifier()         // table.Identifier
tbl.Location()           // string
tbl.MetadataLocation()   // string (path of current metadata.json)
tbl.Metadata()           // table.Metadata
tbl.Schema()             // *iceberg.Schema (current)
tbl.Schemas()            // map[int]*iceberg.Schema
tbl.Spec()               // iceberg.PartitionSpec
tbl.SortOrder()          // table.SortOrder
tbl.Properties()         // iceberg.Properties

if snap := tbl.CurrentSnapshot(); snap != nil {
    fmt.Println(snap.SnapshotID, snap.TimestampMs, snap.Summary)
}

// All snapshots
for _, snap := range tbl.Metadata().Snapshots() {
    fmt.Println(snap.SnapshotID)
}

// Stream all manifest files across all snapshots
for mf, err := range tbl.AllManifests(ctx) {
    if err != nil { /* ... */ }
    fmt.Println(mf.FilePath())
}

Reading data

(t Table) Scan(opts ...ScanOption) *Scan returns a scan that you can resolve into Arrow data.

Streaming record batches

import "github.com/apache/iceberg-go/table"

scan := tbl.Scan()
arrowSchema, batches, err := scan.ToArrowRecords(ctx)
if err != nil { /* ... */ }

fmt.Println(arrowSchema)
for batch, err := range batches {
    if err != nil { /* ... */ }
    fmt.Printf("batch with %d rows\n", batch.NumRows())
    batch.Release()
}

ToArrowRecords returns iter.Seq2[arrow.RecordBatch, error] so only one batch is in memory at a time. Always call batch.Release() to free Arrow buffers.

Materializing as an Arrow Table

arrowTbl, err := tbl.Scan().ToArrowTable(ctx)
if err != nil { /* ... */ }
defer arrowTbl.Release()
fmt.Printf("%d rows in %d cols\n", arrowTbl.NumRows(), arrowTbl.NumCols())

Projection and filters

import "github.com/apache/iceberg-go"

scan := tbl.Scan(
    table.WithSelectedFields("id", "name"),
    table.WithRowFilter(
        iceberg.NewAnd(
            iceberg.GreaterThanEqual(iceberg.Reference("id"), int64(100)),
            iceberg.IsIn(iceberg.Reference("region"), "us-east", "us-west"),
        ),
    ),
    table.WithLimit(1000),
    table.WithCaseSensitive(true),
)

For the predicate vocabulary, see Row Filter Syntax.

Time travel

// By snapshot ID
scan := tbl.Scan(table.WithSnapshotID(snap.SnapshotID))

// As of a timestamp (milliseconds since epoch)
scan = tbl.Scan(table.WithSnapshotAsOf(time.Now().Add(-24*time.Hour).UnixMilli()))

Reading from a branch or tag

scan, err := tbl.Scan().UseRef("audit-branch")
if err != nil { /* ... */ }

arrowTbl, err := scan.ToArrowTable(ctx)

Iterating tasks for custom processing

If you need finer control (custom file readers, distributed scan planning):

scan := tbl.Scan(table.WithRowFilter(myFilter))
tasks, err := scan.PlanFiles(ctx)
if err != nil { /* ... */ }

arrowSchema, batches, err := scan.ReadTasks(ctx, tasks)

Incremental changelog scans

NewIncrementalChangelogScan returns rows added or deleted between snapshots. PlanFiles returns the tasks. Read reads a planned task list, and ToArrowRecords plans and reads. Remote planning is not supported.

schema, batches, err := tbl.NewIncrementalChangelogScan(
    table.WithSelectedFields("id", "name"),
).FromSnapshotExclusive(startSnapshotID).ToArrowRecords(ctx)

FromSnapshotInclusive, FromSnapshotExclusive, and ToSnapshot bound the snapshot range. Projection, row filters, and limits apply while the rows are read. If the start snapshot has been expired, changes in expired snapshots before the oldest retained one are not reported. Each snapshot that adds deletes is planned against every live data file, narrowed only by the partition filter, so the cost follows the table size rather than the size of the snapshot range.

Each batch is the scan projection followed by three columns:

• _change_type — INSERT or DELETE
• _change_ordinal — int32 ordinal of the snapshot that produced the change
• _commit_snapshot_id — snapshot ID that committed the change

Added data files are inserts. Removed data files are deletes of the rows in that file. Delete files added in the range emit the rows they newly remove. A rewritten file emits a DELETE and an INSERT for every row in that file, including rows that did not change. UPDATE rows are not produced. Delete-file handling is iceberg-go behavior.

Writing data

The shortcut methods on Table open a transaction, perform the write, and commit. Use NewTransaction directly when you need to combine multiple operations.

Append

import (
    "github.com/apache/arrow-go/v18/arrow/array"
)

// From a streaming RecordReader
newTbl, err := tbl.Append(ctx, recordReader, nil /* snapshot props */)

// From an in-memory Arrow Table; batchSize controls the rolling writer
newTbl, err = tbl.AppendTable(ctx, arrowTbl, 1024, nil)

Overwrite

import "github.com/apache/iceberg-go/table"

// Replace all data
newTbl, err := tbl.Overwrite(ctx, recordReader, nil)

// Replace only rows matching a filter
newTbl, err = tbl.Overwrite(ctx, recordReader, nil,
    table.WithOverwriteFilter(
        iceberg.EqualTo(iceberg.Reference("date"), "2026-01-01"),
    ),
)

OverwriteTable is the arrow.Table variant.

Delete

newTbl, err := tbl.Delete(ctx,
    iceberg.LessThan(iceberg.Reference("id"), int64(100)),
    nil, /* snapshot props */
)

Add existing files

When you already have data files (e.g. produced by another writer), register them without rewriting:

txn := tbl.NewTransaction()
err := txn.AddDataFilesByPath(ctx, []string{
    "s3://my-bucket/sales/orders/data/file-1.parquet",
    "s3://my-bucket/sales/orders/data/file-2.parquet",
}, nil /* snapshot props */, false /* ignoreDuplicates */)
if err != nil { /* ... */ }

newTbl, err := txn.Commit(ctx)

ReplaceDataFilesByPath(ctx, filesToDelete, filesToAdd, snapshotProps) and ReplaceDataFiles(ctx, filesToDelete, dataFilesToAdd, snapshotProps, opts...) are also available on *Transaction for swapping files atomically.

Transactions

Group writes and metadata changes into one atomic snapshot:

txn := tbl.NewTransaction()

if err := txn.Delete(ctx,
    iceberg.LessThan(iceberg.Reference("date"), "2026-01-01"), nil); err != nil {
    /* ... */
}
if err := txn.Append(ctx, recordReader, nil); err != nil {
    /* ... */
}
if err := txn.SetProperties(iceberg.Properties{"commit.user": "data-pipeline"}); err != nil {
    /* ... */
}

newTbl, err := txn.Commit(ctx)

To target a specific branch:

txn, err := tbl.NewTransactionOnBranch("staging")
if err != nil { /* ... */ }

Commit can retry on conflict (ErrCommitFailed) when commit.retry.num-retries is set. See Automatic retry for the commit.retry.* table properties.

Schema and partition evolution

Schema evolution

import "github.com/apache/iceberg-go/table"

txn := tbl.NewTransaction()

err := table.NewUpdateSchema(txn, true /* caseSensitive */, false /* allowIncompatibleChanges */).
    AddColumn([]string{"tip"}, iceberg.PrimitiveTypes.Float64, "Tip in dollars", false, nil).
    RenameColumn([]string{"name"}, "full_name").
    DeleteColumn([]string{"deprecated_field"}).
    Commit()
if err != nil { /* ... */ }

newTbl, err := txn.Commit(ctx)

Reorder fields with MoveFirst, MoveBefore, or MoveAfter. Set allowIncompatibleChanges to true to permit type narrowing or making optional columns required.

Partition evolution

us := table.NewUpdateSpec(txn, true /* caseSensitive */)
us.AddField("event_time", iceberg.DayTransform{}, "event_day") // sourceColName, transform, partitionFieldName
us.RemoveField("legacy_partition")
if err := us.Commit(); err != nil { /* ... */ }

AddField chains; AddIdentity(sourceCol) is a shortcut for an identity transform; RenameField(name, newName) renames an existing partition field.

Available transforms (root iceberg package): IdentityTransform{}, YearTransform{}, MonthTransform{}, DayTransform{}, HourTransform{}, BucketTransform{NumBuckets: N}, TruncateTransform{Width: W}.

Snapshots and refs

Inspecting

if snap := tbl.CurrentSnapshot(); snap != nil {
    fmt.Println(snap.SnapshotID, snap.TimestampMs, snap.Summary)
}
snap := tbl.SnapshotByID(snapshotID)
named := tbl.SnapshotByName("audit")

for _, s := range tbl.Metadata().Snapshots() {
    fmt.Println(s.SnapshotID, s.Summary)
}

Branches and tags

The CLI's branch create and tag create commands (CLI) are the most ergonomic surface today. Programmatically, ref creation goes through Catalog.CommitTable with a SetSnapshotRef update:

import "github.com/apache/iceberg-go/table"

snap := tbl.CurrentSnapshot()
update := table.NewSetSnapshotRefUpdate(
    "audit",                  // ref name
    snap.SnapshotID,
    table.BranchRef,          // or table.TagRef
    0,                        // maxRefAgeMs (0 = unset)
    0,                        // maxSnapshotAgeMs (0 = unset)
    0,                        // minSnapshotsToKeep (0 = unset)
)
reqs := []table.Requirement{
    table.AssertTableUUID(tbl.Metadata().TableUUID()),
    table.AssertRefSnapshotID("audit", nil), // ref must not already exist
}

_, _, err := cat.CommitTable(ctx, tbl.Identifier(), reqs, []table.Update{update})

Constants table.MainBranch, table.BranchRef, table.TagRef live in table/refs.go. A higher-level builder is on the roadmap.

Expiration and rollback

// Expire snapshots older than the table's retention properties
err := txn.ExpireSnapshots(/* options */)

// Roll back to a previous snapshot
err = txn.RollbackToSnapshot(targetSnapshotID)

Tune retention with the history.expire.min-snapshots-to-keep, history.expire.max-snapshot-age-ms, and history.expire.max-ref-age-ms table properties (see Configuration).

Snapshot integrity audit (v1 and later)

Use ValidateSnapshot to inspect a snapshot's manifest list and entries without changing table data. This is especially useful for checking imported v1 tables before allowing them into production.

report, err := tbl.Inspect().ValidateSnapshot(ctx, snapshotID)
if err != nil {
    return err // I/O failed or the scan was canceled; report may be incomplete.
}
if !report.Valid() {
    for _, issue := range report.Issues {
        fmt.Printf("%s: %s (%s)\n", issue.Code, issue.FilePath, issue.ManifestPath)
    }
}

The report checks manifest versions, partition spec references, content types, manifest entry counts, duplicate manifest paths, and duplicate live content files across all manifests in a snapshot. It also checks that live deletion vectors have a target path, that their blob ranges fit within the recorded file size, and that no two distinct vectors target the same data file. Deleted entries do not count as live. A report is valid only when Complete is true and Issues is empty. This is a metadata-only audit: it does not check data-file contents or whether data-file objects still exist. The audit works with both v1 manifest lists and older v1 snapshots containing embedded manifests paths. For embedded manifests, the partition spec ID is checked against table metadata when the optional header provides it, or inferred when the partition fields match exactly one historical spec. Missing, ambiguous, or mismatched partition specs are reported as issues rather than silently accepted.

The file uniqueness and manifest requirements come from the Iceberg spec and manifest list spec.

Maintenance

Orphan file cleanup

import (
    "time"

    "github.com/apache/iceberg-go/table/maintenance"
)

result, err := maintenance.DeleteOrphanFiles(ctx, tbl,
    maintenance.WithCleanupFilesOlderThan(72*time.Hour),
    maintenance.WithCleanupDryRun(false),
    maintenance.WithCleanupMaxConcurrency(8),
)
if err != nil { /* ... */ }
fmt.Printf("removed %d files\n", len(result.DeletedFiles))

Also see maintenance.WithCleanupLocation, maintenance.WithCleanupDeleteFunc, maintenance.WithPrefixMismatchMode, maintenance.WithEqualSchemes, and maintenance.WithEqualAuthorities in table/maintenance/orphan_cleanup.go.

Compaction (rewrite data files)

import "github.com/apache/iceberg-go/table"

txn := tbl.NewTransaction()
result, err := txn.RewriteDataFiles(ctx, groups /* []table.CompactionTaskGroup */, table.RewriteDataFilesOptions{})
if err != nil { /* ... */ }

newTbl, err := txn.Commit(ctx)
fmt.Printf("rewrote %d files into %d (%d -> %d bytes)\n",
    result.RemovedDataFiles, result.AddedDataFiles, result.BytesBefore, result.BytesAfter)

The table/compaction subpackage provides bin-packing planning. The iceberg compact analyze and compact run CLI commands wrap the same machinery - see CLI.

Compaction tasks represent complete data files, not independent scan splits. The planner collapses split tasks for the same file into one whole-file task, rejecting inconsistent delete or lineage metadata and dropping residual filters because they select files but do not filter rows during compaction. Direct calls to ExecuteCompactionGroup reject partial ranges and residual row filters. Attached delete files are still applied when rewriting each source file.

To sort instead of bin-pack, pass table.WithCompactionSortOrder as a group option. Each group is sorted as a whole, so every output file holds a sorted, non-overlapping range of it. Output files record the order's sort_order_id when the order is one of the table's sort orders, every field has one source column, every field but the last is identity and the last preserves order (see table.WithCompactionSortOrder). Files of different groups still overlap, and by default a group is packed to one target file, so raise compaction.Config.MaxGroupSizeBytes until a group covers a partition. Set compaction.Config.RewriteAll to also rewrite files that are already well-sized:

cfg := compaction.DefaultConfig()
cfg.RewriteAll = true
cfg.MaxGroupSizeBytes = 8 << 30 // 8 GiB per group
plan, err := compaction.Analyze(ctx, tbl, cfg)
if err != nil { /* ... */ }

result, err := txn.RewriteDataFiles(ctx, groups /* from plan.Groups */, table.RewriteDataFilesOptions{
    GroupOptions: []table.CompactionGroupOption{table.WithCompactionSortOrder(tbl.SortOrder())},
})

When every file of a group already records the order's sort_order_id (for example, files written by an earlier sort rewrite), every sort field is identity, and no partition of the group has more files than the scan concurrency (table.WithCompactionScanConcurrency, GOMAXPROCS by default), the files are streamed and merged, holding up to about five batches per file instead of the whole group. Otherwise a sort rewrite holds each group decoded in memory, and with the batches queued for the writer that can reach about twice the group's decoded size, so keep compaction.Config.MaxGroupSizeBytes within what one rewrite can hold. The first sort rewrite of a table always takes this in-memory path, since its files do not record the order yet. The CLI equivalent is iceberg compact run --sort [--rewrite-all] [--max-group-size BYTES], which uses the table's default sort order.

Expiring snapshots

import (
    "time"

    "github.com/apache/iceberg-go/table"
)

txn := tbl.NewTransaction()
err := txn.ExpireSnapshots(
    table.WithOlderThan(7*24*time.Hour),
    table.WithRetainLast(10),
)
if err != nil { /* ... */ }

newTbl, err := txn.Commit(ctx)

Pass table.WithPostCommit(true) to delete the unreferenced data and metadata files after the commit lands. The iceberg expire-snapshots CLI command wraps the same operation - see CLI.

Views

Views are created and loaded through catalogs that support them (REST, Hive, SQL):

import "github.com/apache/iceberg-go/view"

// Create
v, err := view.CreateView(
    ctx,
    "my-catalog",
    table.Identifier{"analytics", "monthly_orders"},
    schema,
    view.Representation{SQL: "SELECT month, sum(amount) FROM orders GROUP BY month", Dialect: "spark"},
    table.Identifier{"sales"},                      // default namespace for unqualified names
    "s3://my-bucket/views/monthly_orders",
    iceberg.Properties{},
)

// Inspect
v.CurrentVersion()      // *view.Version
v.CurrentSchema()       // *iceberg.Schema
v.Versions()            // []*view.Version
v.Schemas()             // map[int]*iceberg.Schema
v.Properties()          // iceberg.Properties

view.New(ident, meta, metadataLocation) constructs a view from already-loaded metadata; view.NewFromLocation(ctx, ident, metadataLocation, fsysFactory) loads metadata from disk or object storage.

Iceberg ↔ Arrow types

When iceberg-go converts an Iceberg schema to Arrow (e.g. for the scanner output) or vice versa, the type mapping is:

Helpers in table/arrow_utils.go:

• SchemaToArrowSchema(sc *iceberg.Schema, nameMapping NameMapping, useLargeTypes, includeRowLineage bool) (*arrow.Schema, error)
• VisitArrowSchema[T](sc *arrow.Schema, visitor ArrowSchemaVisitor[T]) (T, error)

For a writer-side schema (Arrow → Iceberg), the scanner and writers handle conversion automatically as long as your Arrow schema is compatible with the table schema.

Variant Type

variant is the Iceberg v3 semi-structured type: a single self-describing column that holds arbitrary JSON-like values (objects, arrays, and scalars) without a fixed schema. iceberg-go supports both non-shredded and shredded variants, and can push predicates down into individual variant fields.

Reading and writing

A variant column is declared with iceberg.VariantType{} in the table schema and maps to the Arrow variant extension type (see API). Non-shredded variant values need no extra configuration: write an Arrow record whose variant column uses the extension type, and read it back the same way.

Build the column with a variant builder and write it like any other Arrow column (the table must be format version 3):

import (
    "fmt"

    "github.com/apache/arrow-go/v18/arrow"
    "github.com/apache/arrow-go/v18/arrow/array"
    "github.com/apache/arrow-go/v18/arrow/extensions"
    "github.com/apache/arrow-go/v18/arrow/memory"
    "github.com/apache/arrow-go/v18/parquet/variant"
    "github.com/apache/iceberg-go"
    "github.com/apache/iceberg-go/table"
)

// tbl was created with table.PropertyFormatVersion = "3" and this schema.
schema := iceberg.NewSchema(0,
    iceberg.NestedField{ID: 1, Name: "payload", Type: iceberg.VariantType{}},
)
arrowSchema, _ := table.SchemaToArrowSchema(schema, nil, true, false)

bldr := extensions.NewVariantBuilder(memory.DefaultAllocator, extensions.NewDefaultVariantType())
defer bldr.Release()

var vb variant.Builder
_ = vb.Append(map[string]any{"x": int64(320), "target": "button-submit"})
val, _ := vb.Build()
bldr.Append(val)

col := bldr.NewArray()
defer col.Release()
rec := array.NewRecordBatch(arrowSchema, []arrow.Array{col}, 1)
defer rec.Release()

tx := tbl.NewTransaction()
arrTable := array.NewTableFromRecords(arrowSchema, []arrow.Record{rec})
defer arrTable.Release()
_ = tx.AppendTable(ctx, arrTable, 1024, nil)
tbl, _ = tx.Commit(ctx)

A normal scan reads it back; the variant column comes back as an *extensions.VariantArray:

result, _ := tbl.Scan().ToArrowTable(ctx)
defer result.Release()

variants := result.Column(0).Data().Chunk(0).(*extensions.VariantArray)
v, _ := variants.Value(0)
fmt.Println(v.Value()) // variant.ObjectValue for the object above

Shredding

Shredding stores the fields of a variant as typed Parquet sub-columns instead of a single opaque blob. This lets scans read and filter a field as a native column rather than decoding every value. Shredding is opt-in and configured with two write properties (see Configuration):

• write.parquet.shred-variants - enable shredding of top-level variant columns (default false).
• write.parquet.variant-inference-buffer-size - rows buffered per file to infer the shredding schema (default 100).

The reader accepts both shredded and non-shredded data; no read-side configuration is required.

Filtering on variant fields

Filter on a field inside a variant with iceberg.Extract, which plugs into the usual predicate builders. See Row Filter Syntax for usage, supported target types, and caveats.

Constraints

• A variant column cannot be a partition source or an identity-transform source.

Row Filter Syntax

Row filters drive predicate pushdown during scans, partition pruning, and row-level deletes. The DSL lives in the root iceberg package - all you need is the import:

import "github.com/apache/iceberg-go"

A column is referenced with iceberg.Reference("column_name"). Predicate constructors return a BooleanExpression (or an UnboundPredicate, which satisfies BooleanExpression) that can be combined with the boolean combinators in the Expression DSL and passed to APIs like table.WithRowFilter(...).

Equality

iceberg.EqualTo(iceberg.Reference("status"), "active")
iceberg.NotEqualTo(iceberg.Reference("retries"), int32(0))

EqualTo[T] and NotEqualTo[T] are generic over LiteralType (bool, int32, int64, float32, float64, string, []byte, plus a few iceberg-specific types). The value's type must be in that set - bare int literals are not, so use int32(0) / int64(0) explicitly. They wrap LiteralPredicate(OpEQ, ...) and LiteralPredicate(OpNEQ, ...) (predicates.go:83-91).

Comparison

iceberg.LessThan(iceberg.Reference("amount"), 100.0)
iceberg.LessThanEqual(iceberg.Reference("amount"), 100.0)
iceberg.GreaterThan(iceberg.Reference("created_at"), int64(1700000000))
iceberg.GreaterThanEqual(iceberg.Reference("score"), int32(50))

Operators: OpLT, OpLTEQ, OpGT, OpGTEQ (exprs.go:47-50). Constructors at predicates.go:98-124.

Set membership

iceberg.IsIn(iceberg.Reference("region"), "us-east", "us-west", "eu-west")
iceberg.NotIn(iceberg.Reference("status"), "deleted", "archived")

IsIn and NotIn are variadic. They return a BooleanExpression (not UnboundPredicate) because the result can simplify automatically:

• Zero values - reduces to AlwaysFalse{} (for IsIn) or AlwaysTrue{} (for NotIn).
• One value - reduces to EqualTo / NotEqualTo.

See predicates.go:55-78.

Null checks

iceberg.IsNull(iceberg.Reference("deleted_at"))
iceberg.NotNull(iceberg.Reference("user_id"))

These wrap UnaryPredicate(OpIsNull, ...) and UnaryPredicate(OpNotNull, ...). Both panic if the term is nil (predicates.go:23-32).

NaN checks (float / double columns only)

iceberg.IsNaN(iceberg.Reference("ratio"))
iceberg.NotNaN(iceberg.Reference("ratio"))

Operators OpIsNan and OpNotNan. Use these instead of EqualTo(..., math.NaN()) - NaN is never equal to itself, so a comparison or In/NotIn with a NaN literal fails to bind with ErrInvalidArgument.

String prefix

iceberg.StartsWith(iceberg.Reference("path"), "/var/log/")
iceberg.NotStartsWith(iceberg.Reference("name"), "tmp_")

Operators OpStartsWith and OpNotStartsWith (exprs.go:53-54). The value must be a string.

Variant extraction

iceberg.Extract(ref, path, typ) builds a term that navigates a member-selector path into a variant column and casts the leaf to typ. The path uses dot ($.user.age) or quoted-bracket ($['user']['age']) notation; array indexing ($.items[0]), wildcards, filters, and recursive descent are rejected at bind time. It is an UnboundTerm, so it plugs into the same predicate builders as iceberg.Reference:

filter := iceberg.EqualTo(
    iceberg.Extract(iceberg.Reference("payload"), "$.user.age", iceberg.PrimitiveTypes.Int64),
    int64(30),
)

The referenced column must be a variant, and a target type is required. Supported target types: bool, int32, int64, float32, float64, date, time, timestamp, timestamptz, timestamp_ns, timestamptz_ns, string, fixed, binary, decimal, uuid.

Caveats:

• Extract predicates prune whole data files by shredded-field bounds (best-effort, and only for shredded fields), then filter row by row. Row-group statistics and bloom filters do not apply to them.
• Extract terms are DSL-only (no string row-filter form) and are not REST-serializable. Under local scan planning (the default) they run as local residuals; with remote scan planning, a filter containing an Extract term fails to serialize rather than being silently dropped.

See Variant Type for shredding.

Constants

iceberg.AlwaysTrue{}
iceberg.AlwaysFalse{}

These satisfy BooleanExpression and short-circuit during expression simplification. Useful as a base case when filters are built dynamically:

filter := iceberg.BooleanExpression(iceberg.AlwaysTrue{})
for _, clause := range userClauses {
    filter = iceberg.NewAnd(filter, clause)
}

Operator reference

The full operator set is the Operation enum at exprs.go:34-62:

Putting it together

A typical filter passed to a scan:

filter := iceberg.NewAnd(
    iceberg.GreaterThanEqual(iceberg.Reference("event_time"), int64(1700000000)),
    iceberg.IsIn(iceberg.Reference("region"), "us-east", "us-west"),
    iceberg.NotNull(iceberg.Reference("user_id")),
)

scan := tbl.Scan(table.WithRowFilter(filter))

For boolean combination, term details, and the lower-level escape hatches (UnaryPredicate, LiteralPredicate, SetPredicate), see Expression DSL.

Expression DSL

This page covers the building blocks behind the row-filter shortcuts in Row Filter Syntax: boolean combinators, terms, and the lower-level predicate constructors.

The DSL lives entirely in the root iceberg package (see exprs.go and predicates.go).

Boolean combinators

iceberg.NewAnd(a, b)              // a AND b
iceberg.NewAnd(a, b, c, d)        // a AND b AND c AND d (variadic)
iceberg.NewOr(a, b)               // a OR b
iceberg.NewOr(a, b, c, d)         // a OR b OR c OR d
iceberg.NewNot(a)                 // NOT a

NewAnd and NewOr accept two required arguments plus a variadic tail (exprs.go:226, exprs.go:287). They simplify automatically:

• NewAnd(x, AlwaysTrue{}) reduces to x.
• NewAnd(x, AlwaysFalse{}) reduces to AlwaysFalse{}.
• NewOr(x, AlwaysFalse{}) reduces to x.
• NewOr(x, AlwaysTrue{}) reduces to AlwaysTrue{}.
• NewNot(NewNot(x)) reduces to x.

Constants

iceberg.AlwaysTrue{}
iceberg.AlwaysFalse{}

Both satisfy BooleanExpression. Use them as the identity element when composing filters dynamically.

Terms

A term is the left-hand side of a predicate. Iceberg-go has two flavors:

• Reference("column_name") - an unbound term that names a column (exprs.go:373). Typing happens at bind time, when the expression is matched against a schema. This is what you almost always want.
• BoundReference - the resolved form, produced by Reference.Bind(schema, caseSensitive) (exprs.go:389). You only encounter these when writing custom expression visitors.
• Extract(ref, path, typ) - a variant sub-path term that navigates a member-selector path ($.a.b or $['a']['b']; no array indexing or wildcards) into a variant column and casts the leaf to typ (variant_extract.go). Use it inside any predicate builder to filter on a field within a variant. See Row Filter Syntax.

The interfaces are:

type Term interface { ... }                        // shared marker
type UnboundTerm interface { Term; ... }           // pre-bind
type BoundTerm interface { Term; Ref() BoundReference; ... }

(exprs.go:317-348)

Predicates

A predicate applies an Operation to one or more terms. BooleanExpression is the shared interface (exprs.go:123).

Convenience builders (recommended)

For all the common shapes, the constructors in predicates.go are the right tool. See Row Filter Syntax for the full list (EqualTo, LessThan, IsIn, IsNull, StartsWith, etc.).

Lower-level escape hatches

When the convenience builders are not enough (custom operators, dynamic operation selection, working with already-typed Literal values), use the predicate constructors directly:

// Unary predicates: IS NULL / NOT NULL / IS NaN / NOT NaN
pred := iceberg.UnaryPredicate(iceberg.OpIsNull, iceberg.Reference("col"))

// Literal predicates: <, <=, >, >=, ==, !=, STARTS WITH, NOT STARTS WITH
lit := iceberg.NewLiteral(int64(42))
pred := iceberg.LiteralPredicate(iceberg.OpEQ, iceberg.Reference("col"), lit)

// Set predicates: IN / NOT IN
lits := []iceberg.Literal{iceberg.NewLiteral("a"), iceberg.NewLiteral("b")}
pred := iceberg.SetPredicate(iceberg.OpIn, iceberg.Reference("col"), lits)

UnaryPredicate lives at exprs.go:534. LiteralPredicate and SetPredicate live in the same file.

Negation

Every BooleanExpression and every Operation knows how to negate itself.

op := iceberg.OpEQ
op.Negate()  // -> OpNEQ

(exprs.go:65-98. OpNot, OpAnd, and OpOr panic on direct negation - negate the wrapping expression instead.)

expr := iceberg.EqualTo(iceberg.Reference("status"), "active")
inverted := expr.Negate()  // equivalent to NotEqualTo(...)

Binding and evaluation

Most user code stops at constructing the unbound expression - the scan pipeline handles binding, projection, and evaluation internally. If you are writing a custom visitor:

• (Reference).Bind(schema, caseSensitive) returns a BoundTerm (exprs.go:389).
• BoundExpressions expose Ref(), Type(), and (for terms) the underlying accessor for evaluating against a StructLike row.

For projection, evaluation, and visitor patterns, see visitors.go and the scan internals in table/scanner.go.

When to reach for what

For the per-operator cookbook, return to Row Filter Syntax.

Concurrent Writes

When multiple writers commit to the same table, iceberg-go uses optimistic concurrency control: every commit is validated against the table state it was built on. If another writer committed first, the commit is rejected rather than silently clobbering their snapshot.

Conflict detection

Each producer (Append, Overwrite, Delete, RowDelta, RewriteDataFiles) runs a set of validators before the commit lands. They check that the files the operation assumed (added, deleted, or filtered) still match the current table state. A failed validation surfaces as one of:

• table.ErrCommitFailed — the base snapshot moved on; the commit can be retried after refreshing (see below).
• table.ErrCommitDiverged — terminal. The base snapshot is no longer on the branch at all, so a retry cannot reconcile the change.
• table.ErrTransactionUnusable — this transaction can't be committed again. If the error also matches ErrCommitFailed, retry in a new transaction from a reloaded table.

import (
    "context"
    "errors"

    "github.com/apache/iceberg-go/table"
)

_, err := txn.Commit(context.Background())
switch {
case errors.Is(err, table.ErrTransactionUnusable) && errors.Is(err, table.ErrCommitFailed):
    // a concurrent change: retry in a new transaction from a reloaded table
case errors.Is(err, table.ErrCommitDiverged), errors.Is(err, table.ErrTransactionUnusable):
    // unrecoverable: rebuild the operation from the latest table
case errors.Is(err, table.ErrCommitFailed):
    // retriable: refresh and try again (the retry loop does this for you)
case errors.Is(err, table.ErrCommitRejected):
    // the catalog wrote nothing: fix the cause, then call Commit again
}

A commit can also fail with table.ErrCommitRejected, which means the catalog rejected it before writing anything, for example an update the catalog can't apply or a REST 400, 401, 403 or 404. The commit is not retried, and the transaction can still be committed.

Isolation levels

Delete and update operations are validated under an isolation level, set per operation through table properties:

serializable rejects the commit if any concurrent snapshot added data matching the operation's filter. snapshot is more permissive — it only rejects when concurrent deletes touch the same files.

Automatic retry

iceberg-go can refresh the table and replay the operation against the latest snapshot between attempts. On each retry the table metadata is reloaded, the producer's validators re-run against the fresh snapshot, and the commit is re-submitted only if it is still valid. RollbackToSnapshot and AssertRefSnapshotID pin their refs, so retries do not rebase those assertions. If a pinned ref moved, the commit fails with ErrCommitFailed instead of retrying. ExpireSnapshots does not pin refs. Retries replan it as described in Snapshot expiration.

Retry is off by default (commit.retry.num-retries is 0). Opt in by setting the retry properties — see Manifest and commit in the configuration reference for the full list:

// Opt in to retries (e.g. 4 attempts after the first) for a contended table.
_, err := cat.CreateTable(ctx, ident, schema,
    catalog.WithProperties(iceberg.Properties{
        "commit.retry.num-retries": "4",
    }),
)

Catalog support: the REST, SQL, Glue, Hive, and Hadoop catalogs report commit conflicts, including requirements that fail because of a concurrent change, as ErrCommitFailed, so the retry loop engages on all of them. A commit on another branch that takes the staged snapshot's sequence number or first-row-id is a conflict too, and the retry renumbers the snapshot.

Only the implicit assertion on the commit branch's head is rebased after a refresh. Other requirements, including the schema, spec, sort order, and pinned refs, must still hold. Otherwise, the commit fails without another attempt and the transaction is unusable.

After a conflict, calling Commit again on the same transaction refreshes the table and replays the staged changes like the next retry attempt, so it can rebase onto a peer's commit even with retries disabled. Each call gets the configured retry budget. A commit carrying delete-file removals that cannot be replayed resubmits only if its branch has not moved. The rejections after a refresh that make the transaction unusable are listed on ErrTransactionUnusable.

A failed commit keeps the files it staged until a later commit lands. A transaction you abandon leaves them for orphan file removal, as one with retries disabled always has. An abandoned manifest rewrite also leaves the manifests that later attempts merged. TableCommit refuses a transaction whose commit failed, since the multi-table path does not refresh. Call Commit to resume it, or build a new transaction.

Snapshot expiration

A retry replays staged changes in order against the refreshed table. It replans each ExpireSnapshots call at its original position, with the original options and cutoff time. The plan retains snapshots that a concurrent commit made reachable, such as rollback targets and new branch heads. It can expire snapshots that a concurrent append moved outside the retention window. A transaction that appends and expires snapshots rebases onto a concurrent append like any other append.

Each attempt requires the refs read by expiries that remove snapshots or refs to be unchanged since loading or refreshing the table. An expiry that removes nothing adds no such requirement. If a concurrent commit moves one of those refs before the attempt succeeds, it causes another conflict. The next attempt replans the expiry. If a concurrent commit creates a ref that the plan did not read, that ref does not cause a conflict. Expiring its snapshot also removes the ref.

• Retries replan an expiry even if staging or an earlier attempt produced no expiry updates.
• If no updates remain, for example because a concurrent commit already expired the same snapshots, Commit returns the refreshed table without committing.
• A transaction containing only expiries with no updates makes no commit attempt and never refreshes or replans.
• Post-commit cleanup deletes files for snapshots expired by the successful attempt, if cleanup is enabled.

A retry fails with ErrTransactionUnusable if GC is disabled after refresh, a staged change no longer applies, or replanning changes whether a non-main commit branch is removed. ErrTransactionUnusable and ErrCommitFailed list these cases and which of them a new transaction can retry.

Metrics Reporting

Iceberg Go implements Iceberg's Metrics Reporting API. After scan planning it can emit a ScanReport (files and manifests considered, scanned, and skipped; bytes read; planning duration), and after a commit a CommitReport (commit attempts, duration, and the added/removed/total file, record, and delete counts). These metrics are otherwise invisible from outside the client; a reporter gives operators a standard way to collect them.

A pluggable metrics.Reporter is the sink that receives those reports.

Opt-in by default

Reporting is strictly opt-in. With no reporter configured the instrumented code paths do no work, and the client emits no logs and no network traffic.

This is a deliberate divergence from Iceberg Java, which defaults to the logging reporter. A library must not start emitting output the caller never asked for, so Iceberg Go defaults to a no-op reporter until you select one. Callers migrating from Java that want reports by default must select the logging reporter explicitly.

Selecting a reporter

Catalog-wide, via a property

Set metrics-reporter-impl to a registered reporter name. Every table loaded from the catalog inherits it. This works for REST-catalog config too, since it is an ordinary catalog property.

cat, err := catalog.Load(ctx, "prod", iceberg.Properties{
    "metrics-reporter-impl": "logging", // built-in: "nop" (default) or "logging"
})

An unrecognized name is an error, so a typo surfaces rather than silently disabling metrics.

Per-scan override

Override the reporter for a single scan with table.WithReporter; it takes precedence over the reporter inherited from the table.

inmem := &metrics.InMemoryReporter{}

scan := tbl.Scan(table.WithReporter(inmem))
// ... run the scan ...

for _, report := range inmem.Reports() {
    if sr, ok := report.(metrics.ScanReport); ok {
        fmt.Println(sr.TableName, sr.Metrics.ResultDataFiles)
    }
}

Built-in reporters

Writing a custom reporter

Implement the two-method Reporter interface and type-switch on the concrete report:

type myReporter struct{}

func (myReporter) Report(ctx context.Context, report metrics.MetricsReport) {
    switch r := report.(type) {
    case metrics.ScanReport:
        // record r.Metrics ...
    case metrics.CommitReport:
        // record r.Metrics ...
    }
}

func (myReporter) Close() error { return nil } // release any held resources

Two contract rules matter:

• Never block or fail the operation. Report is called inline at the scan/commit completion point. A network-backed reporter must dispatch the send on a background worker, and any error must be handled internally (logged and swallowed), never returned to the caller.
• Be safe for concurrent use. Report may be called from multiple goroutines.

To make a custom reporter selectable by name from metrics-reporter-impl, register a factory for it (typically from init):

func init() {
    metrics.Register("my-reporter", func(props map[string]string) (metrics.Reporter, error) {
        return myReporter{}, nil
    })
}

Reporting to a REST catalog

A REST catalog can also POST each report to the catalog's .../tables/{table}/metrics endpoint. This is a separate opt-in from the in-process reporter above and is off by default; the POST is dispatched on a background worker so it never stalls a scan or commit. See the REST metrics-reporting properties for the enablement flag and timeout.

OpenTelemetry

Experimental. The OpenTelemetry metric and attribute names track the still-unmerged Iceberg Java reporter (apache/iceberg#16250) and may change to stay aligned with it.

The metrics/otel package provides an OpenTelemetry-backed reporter. The host owns the OpenTelemetry SDK; the reporter only looks up a meter (the global MeterProvider by default, or one you pass with WithMeter).

import "github.com/apache/iceberg-go/metrics/otel"

rep, err := otel.NewReporter(
    otel.WithMeter(meter),
    otel.WithAttributes(otel.AttrTableName, otel.AttrOperation),
)

It exports a curated subset of the report fields under iceberg.scan.* / iceberg.commit.* instrument names (e.g. iceberg.scan.planning.duration, iceberg.scan.result.data_files, iceberg.commit.duration, iceberg.commit.data_files.added).

WithAttributes bounds label cardinality by allowlist. The default set is table name + operation; schema id (otel.AttrSchemaID) is opt-in. The snapshot id is never attached as a metric attribute because it is unbounded — per-snapshot detail stays available through the full reports and the table's snapshot history.

The OpenTelemetry reporter is not registered under a metrics-reporter-impl name automatically. To select it by property, register a factory for it as shown in Writing a custom reporter; otherwise pass the instance to table.WithReporter or table.WithMetricsReporter.

Feature Status

This page tracks what Apache Iceberg Go currently supports. The matrix is kept in sync with the project README.md; if you spot a discrepancy, file an issue.

Spec format version coverage

Apache Iceberg Go reads and writes table format versions 1, 2, and 3. The maximum supported version is enforced at table/metadata.go (supportedTableFormatVersion = 3). For active tracking, see issues #589 (V3) and #829 (V2 completion).

V1

All V1 features are supported. V1 is the format-version baseline.

V2

V3

FileSystem support

S3, GCS, and Azure each require a blank import of their backend package, for example _ "github.com/apache/iceberg-go/io/gocloud/s3". See Configuration.

Metadata operations

Catalog support

A Hadoop catalog is also available - see catalog/hadoop.

Read / write data

Data can be read as an Arrow Table or as a stream of Arrow record batches via iter.Seq2. See API Reference.

Supported write operations

As long as the FileSystem is supported and the Catalog supports altering the table:

Metrics reporting

Iceberg's Metrics Reporting API is supported. Reporting is opt-in (default no-op). See Metrics Reporting.

Contributing

Get in Touch

• Iceberg community
• Iceberg-Go Slack

Picking Up Issues

Before starting work on an issue:

1. Check for existing PRs. Search the open pull requests to make sure nobody is already working on it.
2. Claim the issue. Leave a comment on the issue (e.g., "I'd like to work on this") and wait for a maintainer to acknowledge before writing code.
3. One at a time for new contributors. If you haven't had a PR merged into iceberg-go yet, please work on one issue at a time. Get it reviewed, address feedback, get it merged — then pick up the next one. This helps us give your work the attention it deserves and avoids wasted effort from overlapping contributions.

If two PRs land for the same issue, we will generally keep the one from the contributor who claimed it first.

Submitting a Pull Request

• Reference the issue number in your PR description (e.g., "Fixes #123").
• Keep PRs focused — one issue per PR.
• Run make test and make lint before pushing. CI runs all of these too, but catching issues locally saves a round-trip. make lint-install will install the linter if you don't have it yet.
• All commits must have a Signed-off-by line (DCO).
• CI checks the exported API against the base branch with apidiff. If your PR changes it incompatibly on purpose, mark the PR title as breaking with a !, for example refactor(io)!: add context to IO.Open. You can run the same check locally with dev/check-api-compat.sh after installing the apidiff version it names.

Code Review

• Maintainers may request changes. This is normal — it doesn't mean the PR is bad, it means we want to get it right.
• Respond to review comments by pushing new commits (don't force-push over reviewed code).
• If your PR has been waiting for review for more than a few days, ping on Slack.

Development Setup

git clone https://github.com/apache/iceberg-go.git
cd iceberg-go
go build ./...
make test # runs all unit tests
make test-race # runs all unit tests with race detector

Integration Tests

Integration tests require Docker and are gated behind a build tag:

docker compose -f internal/recipe/docker-compose.yml up -d rest object-store create-bucket --wait
go test -tags integration ./...

External REST Catalog Conformance

TestExternalRestCatalogConformance runs the catalog conformance suite against any REST catalog server. It is skipped unless REST_CATALOG_URI is set. CI runs it against Apache Polaris; to do the same locally (requires jq):

make integration-setup-polaris
make integration-external-rest

To test another catalog, override the Polaris defaults with REST_CATALOG_URI, REST_CATALOG_CREDENTIAL (client_id:client_secret), REST_CATALOG_SCOPE and REST_CATALOG_WAREHOUSE. Set any of them to an empty value to leave it unset, for example:

REST_CATALOG_URI=http://localhost:8181 REST_CATALOG_CREDENTIAL= REST_CATALOG_SCOPE= REST_CATALOG_WAREHOUSE= \
  make integration-external-rest

Community

Apache Iceberg Go is developed in the open as part of the broader Apache Iceberg project. The fastest ways to ask questions, share work, and follow development are below.

Chat

• #iceberg-go on Apache Iceberg Slack - day-to-day discussion, design questions, help requests.

Mailing list

The Apache Iceberg project uses a single dev mailing list across all language implementations.

• Read / post: dev@iceberg.apache.org
• Subscribe: send any email to dev-subscribe@iceberg.apache.org

Issues and pull requests

• Issue tracker
• Pull requests

For contribution guidelines, see Contributing.

Apache Iceberg community at large

• Apache Iceberg community page - cross-implementation links, sync calls, and the project Code of Conduct.
• Apache Iceberg Code of Conduct - applies to all participation in this project.

Glossary

This glossary defines important terms used throughout the Iceberg ecosystem, organized in tables for easy reference.

Core Concepts

Data Types

Primitive Types

Nested Types

Operations

Partitioning

Partition Transforms

Expressions and Predicates

File Formats

Metadata

Transactions

References

Storage

Query Optimization

Schema Evolution

Time Travel

ACID Properties

Releases

Apache Iceberg Go follows the standard Apache release process. Releases are cut from main, voted on by the Apache Iceberg PMC, and published as signed source tarballs to https://downloads.apache.org/iceberg/ and as Go module versions tagged vX.Y.Z.

Latest release

• Latest release on GitHub
• All releases: github.com/apache/iceberg-go/releases

Using a release

Pin a tagged version directly with Go modules:

go get github.com/apache/iceberg-go@vX.Y.Z

To track main, use @main instead of a version tag.

Release notes

Per-release notes (highlights, breaking changes, contributors) are published on the GitHub Releases page. Iceberg Go does not maintain a curated CHANGELOG.md in the repository; the GitHub Releases page is the canonical source.

Unreleased compatibility notes

• Schemas store binary and fixed defaults as hex strings. NewSchema, NewSchemaWithIdentifiers and UpdateSchema.AddColumn (through Apply()) convert []byte, BinaryLiteral and FixedLiteral defaults of binary and fixed fields, including fields nested in structs, lists and maps, to the lowercase hex string iceberg-go writes to metadata JSON, so a field read back from a schema holds a string such as "0aff". Code that type-asserts these defaults to []byte must decode the string instead, for example with hex.DecodeString. A schema built in process now equals the same schema read from metadata with lowercase hex. Equals compares the strings case-sensitively, and Java writes uppercase hex (see apache/iceberg#18352). The fields and types passed in are not modified.

• Predicates reject NaN literals when bound. EqualTo, NotEqualTo, LessThan, LessThanEqual, GreaterThan, GreaterThanEqual, IsIn, NotIn and LiteralPredicate/SetPredicate with a NaN float literal now fail in Bind with ErrInvalidArgument, as Java does. That includes the string "NaN" on a float column and a NaN on a decimal column, which used to bind as zero. Scans, ExpressionEvaluator and the metrics evaluators all bind first, so they return the error too. Use IsNaN and NotNaN to match NaN values.

• SQL view APIs align with catalog.ViewCatalog. CreateView now takes a *view.Version, schema and options, and returns *view.View. LoadView now returns *view.View; use Metadata() to access its metadata. If the caller leaves DefaultCatalog empty, SQL still sets it to the catalog name.

• LocationProvider.NewDataLocation takes the partition path. The method is now NewDataLocation(dataFileName, partitionPath string), matching Iceberg Java and PyIceberg, so custom LocationProvider implementations must add the argument. Callers pass the partition path there, or "" for an unpartitioned file, instead of prefixing the file name.

• Partitioned data files follow the table's location provider. With write.object-storage.enabled=true and the default write.object-storage.partitioned-paths=true, new partitioned files land under data/<hash>/<partition>/ instead of data/<partition>/<hash>/, which can affect prefix-based lifecycle rules and bucket policies. write.object-storage.partitioned-paths=false and write.data.path now apply to partitioned writes. Partitioned paths also no longer percent-encode the table location: s3://bucket/my table stays literal instead of becoming my%20table, as for unpartitioned writes. Existing files stay where they are.

• Catalog capabilities split into narrower interfaces. catalog.Catalog now embeds TableCatalog and NamespaceCatalog with unchanged table and namespace methods. Name() string is now required, breaking third-party catalogs and test fakes. ViewCatalog is optional (REST, Hive, and SQL), while REST also implements UpdatableViewCatalog for UpdateView. Use catalog.As[T](cat) and the Unwrap() Catalog convention to discover optional capabilities through decorators. Future capabilities should use optional interfaces rather than extending Catalog.

• ReplaceFilesWithDeleteFiles removed. For delete-file replacements, use tx.NewRewrite(snapshotProps), call DeleteFile for each source file and AddDeleteFileWithDataSequenceNumber for each replacement, then commit the builder. Rewrites record a replace snapshot instead of overwrite. The five-argument ReplaceFiles API is unchanged and remains available for compaction and delete-file removals.

• Manifest and deletion-vector reads take a context. Snapshot.Manifests, ManifestFile.Entries, iceberg.EntriesWithProjection, dv.ReadDV and dv.ReadDVs now take a context.Context first argument and pass it to IO.Open. Custom ManifestFile implementations must add it to Entries.

• Path-based data-file addition renamed. Replace Transaction.AddFiles with AddDataFilesByPath, AddFilesOption with AddDataFilesByPathOption, and WithAddFilesConcurrency with WithAddDataFilesByPathConcurrency. Behavior is unchanged.

• Manifest builder constructor renamed. Replace iceberg.NewManifestFile(...) with iceberg.NewManifestBuilder(...). It still returns a ManifestBuilder.

• Manifest sequence APIs use explicit names. ManifestBuilder.SequenceNum and ManifestFile.SequenceNum become SequenceNumber, and ManifestFile.MinSequenceNum becomes MinSequenceNumber. ManifestEntryBuilder.SequenceNum and ManifestEntry.SequenceNum become DataSequenceNumber. FileSequenceNum becomes FileSequenceNumber for entries and their builder. Avro and JSON field names are unchanged.

• View and UDF metadata builder constructors are infallible. view.NewMetadataBuilder and udf.NewMetadataBuilder now return only a builder instead of (builder, error); remove error handling at call sites.

• Table metadata getter renamed. Replace table.Metadata.Version() with table.Metadata.FormatVersion(), consistent with Iceberg Java and Go view/UDF metadata. Metadata JSON is unchanged.

• Snapshot ref type field renamed. Replace table.SnapshotRef.SnapshotRefType with table.SnapshotRef.Type. The JSON field is still "type".

• As-of snapshot lookups now use inclusive timestamps. Replace tbl.SnapshotAsOf(timestampMs, true) with tbl.SnapshotAsOf(timestampMs). The method always selects the latest snapshot at or before the timestamp, matching Java Iceberg's SnapshotUtil.nullableSnapshotIdAsOfTime. For strict-before lookup, use SnapshotLogEntryAsOf(tbl.Metadata().SnapshotLogs(), timestampMs, false).

• Snapshot timestamp property key corrected. PropertyCurrentSnapshotTimestamp now uses current-snapshot-timestamp-ms. The old current-snapshot-timestamp spelling is no longer reserved and remains usable as a custom property.

• FileIO open and remove take a context. io.IO.Open and io.IO.Remove now take a context.Context first argument, so custom io.IO implementations must add it. maintenance.WithCleanupDeleteFunc now takes func(context.Context, string) error.

• Hadoop catalog construction takes a context. hadoop.NewCatalog(name, warehouse, props) becomes hadoop.NewCatalog(ctx, name, warehouse, props). The context bounds filesystem setup; the catalog's filesystem keeps its values but not its cancellation.

• Config loading reports errors. config.LoadConfig now returns ([]byte, error), and config.LoadConfigFile is removed; call LoadConfig instead. A missing default ~/.iceberg-go.yaml still returns no data and no error.

• Branch transactions return initialization errors. Table.NewTransactionOnBranch(branch) now returns (*Transaction, error), and Table.NewTransactionOnBranchWithError is removed. Use NewTransactionOnBranch and handle its error.

• Snapshot expiration now tolerates concurrent ref changes. If a concurrent commit moves a ref that the expiry read, a retry or later Commit replans instead of failing with table.ErrCommitFailed. The committed expiry can differ from Transaction.StagedTable. See Snapshot expiration for cases that now fail with table.ErrTransactionUnusable.

• Schema JSON constructor uses the Go initialism. Replace iceberg.NewSchemaFromJsonFields with iceberg.NewSchemaFromJSONFields.

• AWS identifiers use the Go initialism. glue.AwsProperties, glue.WithAwsConfig and glue.WithAwsProperties become glue.AWSProperties, glue.WithAWSConfig and glue.WithAWSProperties; sigv4.WithAwsConfig becomes sigv4.WithAWSConfig; utils.WithAwsConfig and utils.GetAwsConfig become utils.WithAWSConfig and utils.GetAWSConfig; and config.CatalogConfig.AwsProfile becomes AWSProfile. The aws-profile YAML key is unchanged.

• KMS factory registration now takes a context. Change encryption.KMSFactory implementations to accept (context.Context, map[string]string) instead of only properties, and call encryption.LoadKeyManagementClient(ctx, props) instead of LoadKeyManagementClient(props). The existing global RegisterKMS, UnregisterKMS, and GetRegisteredKMSNames functions still use the default registry; RegisterKMS still panics for invalid registrations. Use encryption.MustRegisterKMS for provider initialization. For isolated registrations, create registry := encryption.NewKMSRegistry() after importing any KMS providers and call registry.Register, registry.Load(ctx, props), registry.Names, or registry.Unregister. New registries snapshot the global registrations; subsequent changes do not propagate.

• Name-mapping child lookup renamed. Replace MappedField.GetField(name) with MappedField.FindFieldByName(name), which returns (*MappedField, bool). The new method returns a pointer to the actual child field rather than a copy, and reports a missing name with false.

• Data file block size setter removed. iceberg.DataFileBuilder.BlockSizeInBytes is removed. New v1 manifests use the Avro schema's 64 MiB default when the field is unset. V2/v3 omit this deprecated field, and legacy v1 values remain readable.

• View creation APIs take a named SQL representation. view.NewVersionFromSQL, view.CreateView, and view.CreateViewWithIOProperties now accept a view.Representation{SQL: sql, Dialect: dialect} instead of positional SQL/dialect strings. The dialect is required; Hive preserves the selected representation's dialect.

• View SQL representation construction uses named fields. Replace view.NewRepresentation(sql, dialect) with view.NewRepresentation(view.Representation{SQL: sql, Dialect: dialect}). Serialized JSON is unchanged.

• Hive views keep the caller's view version. hive.Catalog.CreateView now stores every representation, the summary, the default namespace and default-catalog as passed, instead of one representation with default-catalog set to "hive".

• Reserved table properties now use a read-only API. Replace references to table.ReservedProperties with table.ReservedProperties(). The function returns a fresh slice, so callers cannot change metadata validation by modifying it.

• Fixed and decimal constructors return errors. iceberg.FixedTypeOf and iceberg.DecimalTypeOf now return (type, error) instead of panicking for invalid inputs. Use MustFixedTypeOf or MustDecimalTypeOf only for known-valid values.

• SQL UDF representations now use struct literals. The udf.NewSQLRepresentation(dialect, sql) function, available since v0.7.0, is removed. Use udf.SQLRepresentation{Dialect: "trino", SQL: "x + 1"} instead. Empty fields and duplicate dialects are checked during metadata validation; serialized JSON is unchanged.

• Filesystem adapter construction consolidated. Replace io.FSPreProcName(fsys, fn) with io.FS(fsys, io.WithNamePreprocessor(fn)). Plain io.FS(fsys) remains supported.

• Table purge is required on catalogs. The catalog.Catalog interface now embeds catalog.PurgeableCatalog; every custom catalog implementation must provide PurgeTable(ctx, identifier) error. The narrow catalog.PurgeableTable interface is renamed to catalog.PurgeableCatalog. Built-in purge behavior is unchanged.

• Unify statistics blob types with Puffin. Replace table.BlobType with puffin.BlobType when constructing table.BlobMetadata. Replace table.BlobTypeApacheDatasketchesThetaV1 with puffin.BlobTypeDataSketchesTheta and table.BlobTypeDeletionVectorV1 with puffin.BlobTypeDeletionVector. Use puffin.BlobType.IsKnown() to identify the types defined by the Puffin specification; the redundant table.BlobType.IsValid() method is removed. Unknown non-empty blob type names still round-trip through table metadata.

• View representation field uses the Go SQL initialism. Replace view.Representation.Sql with view.Representation.SQL. Serialized view metadata still uses the sql JSON field.

• Orphan cleanup results now expose only OrphanFiles. Replace result.OrphanFileLocations with result.OrphanFiles (each entry has Path and SizeBytes). OrphanCleanupPlan.Files() still provides path-only lists during planning.

• Catalog identifier helper renamed. catalog.TableNameFromIdent is removed. Use catalog.ObjectNameFromIdent for tables, views, and functions.

• Substrait expression conversion no longer takes case sensitivity. substrait.ConvertExpr(schema, expression, caseSensitive) becomes substrait.ConvertExpr(schema, expression). Case sensitivity remains part of expression binding.

• Transform formatting requires the result type and uses the Java-aligned method name. Transform.ToHumanStr(value) and Transform.ToHumanStrType(resultType, value) are removed. Use Transform.ToHumanString(resultType, value) to format partition values, including timestamps with time zones.

• Catalogs now require cleanup. The catalog.Catalog interface includes Close() error. Custom catalog implementations must provide Close, and callers should close catalog instances after use. catalog.Closer is removed. Replace cat.(catalog.Closer) checks with defer cat.Close(); catalog.Catalog already implements io.Closer, so no type assertion is needed.

• Puffin writer property setter renamed. puffin.Writer.AddProperties is now puffin.Writer.SetProperties. Repeated calls still merge keys; use ClearProperties to clear custom properties before finishing.

• Orphan file cleanup moved to table/maintenance. The orphan-cleanup and file-purge API is no longer on table.Table; it now lives in the table/maintenance package as free functions that take the table as their first argument (matching table/compaction). For example, tbl.DeleteOrphanFiles(ctx, ...) becomes maintenance.DeleteOrphanFiles(ctx, tbl, ...). Because table/maintenance imports table (and not the reverse), this cannot be offered as a deprecation shim on table.Table without an import cycle, so callers must switch to the new package. The following moved from table to table/maintenance:

  • Functions (now free functions taking *table.Table): DeleteOrphanFiles, PlanOrphanFiles, ExecuteOrphanCleanup, PurgeFiles.
  • Types: OrphanCleanupOption, OrphanCleanupResult, OrphanFile, OrphanCleanupPlan, PrefixMismatchMode.
  • Constants: PrefixMismatchError, PrefixMismatchIgnore, PrefixMismatchDelete.
  • Option constructors: WithCleanupLocation, WithCleanupFilesOlderThan, WithCleanupDryRun, WithCleanupDeleteFunc, WithCleanupMaxConcurrency, WithPrefixMismatchMode, WithEqualSchemes, WithEqualAuthorities. The four generic names (WithLocation, WithFilesOlderThan, WithDryRun, WithDeleteFunc) gained the WithCleanup prefix for consistency.

  Relatedly, the previously unexported isGCEnabled is now table.IsGCEnabled, so the new package can gate destructive operations on the gc.enabled table property.

• Data-file replacement methods renamed. Use ReplaceDataFilesByPath for string paths. Use ReplaceDataFiles for prebuilt DataFile values instead of ReplaceDataFilesWithDataFiles.

• maintenance.PurgeFiles (previously Table.PurgeFiles) now calls IO.Remove concurrently, with up to 32 calls in flight, when the filesystem does not implement BulkRemovableIO. Custom IO implementations, including those registered with io.Register, must make Remove safe for concurrent use and synchronize any shared mutable state. The bulk deletion path is unchanged.

• Snapshot summaries report deletion vectors separately. On v3 tables, DVs are counted under added-dvs and removed-dvs instead of added-position-delete-files and removed-position-delete-files. added-files-size and removed-files-size use each DV's content_size_in_bytes instead of its whole Puffin file, so they go down for commits with DVs. total-files-size is updated from these values. Partition summaries, commit reports and OTel metrics change the same way. Existing snapshots are not rewritten.

• Partition and sort source accessors renamed. Replace PartitionField.SourceID() and SortField.SourceID() with FirstSourceID(). Their behavior and serialized fields are unchanged.

• Row-lineage column predicate renamed. Use iceberg.IsRowLineageColumn instead of iceberg.IsMetadataColumn to identify _row_id and _last_updated_sequence_number. Use iceberg.IsReservedFieldID to check the full reserved metadata field ID range.

• File writes and directory walks now take an operation context. Update WriteFileIO and ListableIO implementations and calls to Create, WriteFile, and WalkDir to pass context.Context; cloud writes and listings use it for cancellation.

• In and NotIn follow SQL semantics for NULL. A multi-valued NotIn, NOT(In(...)) and NOT(NotIn(...)) no longer match rows where the column is NULL, consistent with NotEqual. A predicate under NOT on a column missing from an older data file no longer matches every row of that file; those rows match as a stored NULL would. Java's Evaluator still matches NULL for notEq and notIn, see apache/iceberg#18701.

• Commit rejections now match table.ErrCommitRejected. SQL, Glue, Hive and Hadoop failures while preparing metadata, plus REST commit responses 400, 401, 403 and 404, leave transactions committable (after a REST 404, the next Commit fails when it reloads the table). Existing errors.Is matches keep working, but SQL, Glue, Hive and Hadoop rejection messages now start with commit rejected: , so code that matches error text may need updating.

• Float comparisons against literals beyond the float32 range follow IEEE 754. On a float32 column, x < 1e100 no longer returns NaN, +Inf or NULL rows, and NOT(x > 1e100) no longer returns NULL rows. An infinite literal now binds to a float32 infinity, so x = +Inf matches +Inf rows. Java still folds these comparisons to always true or false, see apache/iceberg#18700.

Verifying and producing releases

If you are validating an RC or cutting a new release, see:

• Verify a release - what to do when a [VOTE] thread is posted on dev@iceberg.apache.org.
• How to release - PMC/committer process for cutting an RC and publishing.

Verify a release

When a release candidate is announced on dev@iceberg.apache.org, anyone (committer or not) can help by verifying the artifacts. Verification is required from at least three Apache Iceberg PMC members for the vote to pass.

The repository ships a script that performs the full verification end-to-end: dev/release/verify_rc.sh.

What an RC announcement contains

A [VOTE] iceberg-go X.Y.Z RC<N> thread on dev@iceberg.apache.org will reference:

• A signed source tarball under https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-go-${VERSION}-rc${RC}/ (.tar.gz, .tar.gz.asc, .tar.gz.sha512)
• The Apache Iceberg KEYS file at https://downloads.apache.org/iceberg/KEYS
• A GitHub compare URL between the previous release tag and the new RC tag

Prerequisites

The script requires:

• curl
• gpg
• shasum or sha512sum
• tar

You do not need Go installed - if Go is not on the system, the latest Go is downloaded automatically and used only for verification.

Import the Apache Iceberg KEYS

curl https://downloads.apache.org/iceberg/KEYS -o KEYS
gpg --import KEYS

Run the verification script

The script takes the version and RC number as positional arguments:

dev/release/verify_rc.sh ${VERSION} ${RC}

For example, to verify 0.6.0 RC1:

dev/release/verify_rc.sh 0.6.0 1

If the verification succeeds, the script prints:

RC looks good!

Optional environment variables

verify_rc.sh honors these environment variables (all optional):

Manual verification fallback

If you would rather verify by hand, the underlying steps are:

# Download the artifacts
curl -O https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-go-${VERSION}-rc${RC}/apache-iceberg-go-${VERSION}.tar.gz
curl -O https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-go-${VERSION}-rc${RC}/apache-iceberg-go-${VERSION}.tar.gz.asc
curl -O https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-go-${VERSION}-rc${RC}/apache-iceberg-go-${VERSION}.tar.gz.sha512

# Verify the signature
gpg --verify apache-iceberg-go-${VERSION}.tar.gz.asc apache-iceberg-go-${VERSION}.tar.gz

# Verify the checksum (or sha512sum -c on systems without shasum)
shasum -a 512 --check apache-iceberg-go-${VERSION}.tar.gz.sha512

# Inspect the contents
tar xf apache-iceberg-go-${VERSION}.tar.gz

You should reply to the [VOTE] thread with +1, +0, or -1 and a one-line description of what you verified (for example, "verified signatures, checksums, and verify_rc.sh passed on macOS arm64 with system Go 1.25.5").

How to release

This page is for Apache Iceberg PMC members and committers who are cutting a new Apache Iceberg Go release. The canonical scripts live in dev/release/ and dev/release/README.md is the source of truth - this page mirrors it and adds context.

Requirements

• You must be an Apache Iceberg committer or PMC member.

• You must prepare a PGP key for signing. See https://infra.apache.org/release-signing.html#generate.

• Your PGP key must be registered in the Apache Iceberg KEYS file at https://downloads.apache.org/iceberg/KEYS. To add a key:

  svn co https://dist.apache.org/repos/dist/release/iceberg
  cd iceberg
  $EDITOR KEYS
  svn ci KEYS
  

• You must run the release scripts from a working copy whose origin remote is git@github.com:apache/iceberg-go.git (not your fork). release_rc.sh enforces this.

Overview

1. Test the revision to be released.
2. Prepare an RC and start a vote.
3. On a passing vote, publish.

Prepare an RC and vote

Run dev/release/release_rc.sh against the canonical clone:

git clone git@github.com:apache/iceberg-go.git
cd iceberg-go
GH_TOKEN=${YOUR_GITHUB_TOKEN} dev/release/release_rc.sh ${VERSION} ${RC}

Example for 0.6.0 RC1:

GH_TOKEN=${YOUR_GITHUB_TOKEN} dev/release/release_rc.sh 0.6.0 1

The arguments are the version and the RC number. If RC1 has a problem, increment the RC number to RC2, RC3, and so on.

release_rc.sh will:

• Tag vX.Y.Z-rc<N> and push the tag.
• Create a signed source tarball.
• Upload the artifacts to https://dist.apache.org/repos/dist/dev/iceberg/.
• Print a draft [VOTE] email you can use on dev@iceberg.apache.org.

Send the [VOTE] email. The vote runs for at least 72 hours and requires three +1 votes from PMC members with no -1 votes to pass.

Publish

When the vote passes, run:

GH_TOKEN=${YOUR_GITHUB_TOKEN} dev/release/release.sh ${VERSION} ${RC}

release.sh moves the artifacts from https://dist.apache.org/repos/dist/dev/iceberg/ to https://dist.apache.org/repos/dist/release/iceberg/ (which feeds https://downloads.apache.org/iceberg/) and creates a GitHub Release with auto-generated notes.

Post-release tasks

After publishing, complete these steps:

1. Add the release to ASF's report database at the Apache Committee Report Helper.
2. Verify the GitHub Release at https://github.com/apache/iceberg-go/releases is correctly tagged, has generated release notes against the prior tag, and is marked as latest. release.sh runs gh release create ... --generate-notes --verify-tag so the release should already exist; double-check the notes and the "Latest" badge.
3. Send the [ANNOUNCE] email to dev@iceberg.apache.org and announce@apache.org.
4. File a release blog post in apache/iceberg under site/docs/blog/posts/. See the prior 0.5.0 post (2026-03-05-iceberg-go-0.5.0-release.md) for the frontmatter and structure.

Patch releases

dev/release/README.md does not document a patch-branch convention (e.g. iceberg-go-0.X.x). Confirm with the PMC on dev@iceberg.apache.org before cutting a patch release.

Other Iceberg implementations

Apache Iceberg Go is one of several official Iceberg implementations. Pick the one that matches your runtime; the table format and catalog protocols are the same across all of them.

When to use which

• Java is the reference implementation and is what every query engine integrates against (Spark, Flink, Trino, Hive, Presto, Dremio, etc.). If you are running a JVM workload, this is the canonical choice.
• PyIceberg is for Python and the dataframe ecosystem (PyArrow, Pandas, Polars, DuckDB, Daft, Ray). Most data-science and ML workflows live here.
• iceberg-rust is the Rust implementation, used by pyiceberg-core, DataFusion-based engines, and other Rust-native systems.
• iceberg-cpp is early-stage (0.2.0 released 2026-01-26). Track the project for native C++ integration once it stabilizes.
• iceberg-go (this project) is for Go services and tooling. Tight Apache Arrow Go integration makes it a good fit for streaming Arrow record batches into and out of Iceberg tables.

Specifications and shared concepts

The Iceberg spec, terminology, partitioning semantics, evolution semantics, REST Catalog OpenAPI, and multi-engine support policy live with the main project at iceberg.apache.org. All the implementations above target the same spec.

For Apache Iceberg Go-specific guidance, continue with the API Reference, CLI, or Configuration.