The Iceberg Catalog Layer on AWS - Catalog Federation, the REST Catalog API, and Scoped Credential Vending

First Published:
Last Updated:

The decision to adopt Apache Iceberg has already been made by many organizations. Tables reside on Amazon S3, the format is defined, and the question of which engine supports which format version has been resolved. However, design discussions still stall in one place: which catalog is to be treated as the one that correctly knows where those tables are.

The challenge lies in the fact that the answer can vary across different teams within an organization. The team that holds the data registers it in its own catalog, and the team that wants to analyze it has a platform built on AWS Glue Data Catalog. Neither side is wrong. Consolidating onto one of them is expensive, politically and operationally. What remains is the option of connecting the two and leaving both in place.

AWS ships this as catalog federation. However, simply following the configuration steps can lead you to miss the most important aspects. The catalog layer settles three separate things: which catalog holds the authoritative metadata for a table, the path that reaches the remote catalog and proves who you are, and who vends the credential for the data that sits on S3. These three sit side by side in one configuration screen, and they operate independently and fail independently.

This article breaks down the catalog layer into these three distinct roles. It is not a step-by-step configuration guide. Rather, the key takeaway for the reader should be a clear understanding of where to locate the definitive metadata for their tables, and a comprehensive list of the limitations that result from that choice. All information presented in this article was verified as of September 5, 2026.

Table of Contents

  1. 1. Introduction - The Decisions This Article Supports
  2. 2. Why the Catalog Became the Layer That Is Argued Over
  3. 3. Three Roles in the Catalog Layer
  4. 4. The Protocol Runs in Both Directions
  5. 5. What a Federated Catalog Actually Does
  6. 6. Reaching the Remote Catalog
  7. 7. The Credential for the Data Is a Separate Decision
  8. 8. What the Engine Receives, and Where It Goes Next
  9. 9. What This Design Takes Away
  10. 10. Where the Primary Sources Disagree
  11. 11. Failure Modes
  12. 12. Frequently Asked Questions
  13. 13. Summary
  14. 14. References

1. Introduction - The Decisions This Article Supports

1.1 The Position This Article Assumes

This article is intended for data infrastructure architects who want to enable access to Iceberg tables (located on top of S3) from multiple AWS engines, as well as from catalogs outside of AWS. The tables already exist, and discussions regarding their format are complete. What remains is that the question of who decides a table's location and permissions is answered differently on each side of the organization.

Specifically, the situation this article assumes is one where an analytics platform resides on AWS, and Amazon Athena, Amazon Redshift, and Amazon EMR all need to access the same table. Simultaneously, the team responsible for generating the data is using a different catalog implementation, which they treat as the authoritative one. They have built their own system to periodically copy metadata between the two catalogs, but this synchronization process is creating an operational burden due to delays and failures.

1.2 What Other Articles Already Cover

This article is limited to the catalog layer. The neighboring topics are already held by existing articles, and the boundary is drawn here first.

1.3 Four Things That All Get Called Catalog

The subject of this article is the word catalog, and inside one article that word can point at four different things. The distinction is made here, at first use, and is kept for the rest of the article.

Term Used in This ArticleRefers To
AWS Glue Data CatalogThe metadata repository itself inside AWS. This is the official name of the service, and it is often abbreviated as "Data Catalog."
Catalog objectA level inside the Data Catalog that holds databases and tables.
Federated catalogA catalog object created inside the Data Catalog that mirrors a catalog object living on a remote catalog server.
Remote catalogAn Iceberg catalog implementation that lives outside AWS and serves an API that follows the Iceberg REST specification.

The same term also turns up in another layer. Metadata inside PostgreSQL is also called a catalog, but this refers to the internal workings of a single relational database, and is distinct from the "catalog" discussed in this article. This area is covered in PostgreSQL Autovacuum, Bloat, and Planner Statistics on Aurora and RDS - Why the Defaults Stop Being Enough, and What to Tune First, which uses the term "system catalog" to differentiate it.

1.4 Three Questions This Article Answers

This article supports three key decisions.

  1. The first is deciding which side holds the authoritative metadata. Should the remote catalog stay authoritative, with AWS reading from it, or should the Data Catalog on the AWS side be authoritative, with external engines reading from it? Both use the same Iceberg REST protocol, but they run in opposite directions, and they break in opposite ways.

  1. The second is whether you design the metadata path and the data path separately. In a federated catalog configuration, metadata resolution and data access can follow entirely different paths. A state in which only one of them is working does occur.

  1. The third is knowing what that choice takes away. Catalog federation removes the need to migrate, and in exchange it pins down a set of assumptions. These include the namespace hierarchy, case sensitivity of identifiers, the size of the metadata, and the property that deletions on the remote side do not impact permissions on this side.

2. Why the Catalog Became the Layer That Is Argued Over

2.1 What the Table Format Settled, and What It Left Open

Apache Iceberg addressed the challenge of maintaining transactional integrity over object storage. Schema evolution, partition evolution, time travel, and the arbitration of concurrent writes were all settled as part of the table format specification, and several engines now implement that same specification.

However, even with the format defined, one crucial question remains: who knows the location of the current metadata file for a given table? An Iceberg table is a chain of metadata files, and every write creates a new one. One pointer has to say which of them is current, and the catalog is what holds that pointer and swaps it atomically.

The catalog is therefore not an accessory to the table format. It is the party that finally guarantees a write is correct. Hold two catalogs, then, and one table can carry two truths. Organizations' reluctance to consolidate catalogs is not simply a matter of migration costs.

2.2 What the Iceberg REST Specification Changed

The Iceberg REST Catalog specification made that party replaceable. Engines no longer need to have a client for each catalog implementation; instead, they only need to implement a single HTTP interface.

The specification defines operations for reading and writing to namespaces, tables, and views. The Apache Iceberg 1.6.1 OpenAPI definition includes 25 operationId values; the core operation for reading tables is loadTable, and the core operation for listing namespaces is listNamespaces. Both names return in later sections, because AWS documentation calls the same operations by other names.

The specification also includes provisions for the exchange of credentials. Through the X-Iceberg-Access-Delegation header, a client tells the server that it can handle delegated access. The specification describes the header this way.

Optional signal to the server that the client supports delegated access via a comma-separated
list of access mechanisms. The server may choose to supply access via any or none of the
requested mechanisms.

The header takes two values. vended-credentials is the mechanism where the server returns temporary credentials, and remote-signing is the one where the server signs on the client's behalf. Importantly, the specification includes a placeholder for credentials, designed to be filled in by the implementation. Who fills that placeholder varies depending on the implementation. On AWS, Lake Formation fulfills that role.

2.3 This Article Does Not Retell the History

The history of when each company released the Iceberg catalog, when they changed their names, and when they implemented support for the Iceberg REST Catalog API is documented in detail, with dates and primary source materials, in the previously published Cloud Data Warehouse History and Timeline. Additional dates specific to AWS are available in AWS History and Timeline regarding AWS Glue. This article does not present a timeline.

To summarize where things stand: the Data Catalog gained an Iceberg REST endpoint on December 3, 2024, and on November 24, 2025, catalog federation to remote Iceberg catalogs reached general availability. The first date appears in the document history for AWS Glue, the second in the What's New announcement. The former refers to the Data Catalog acting as the endpoint that responds as a REST catalog, while the latter refers to the Data Catalog acting as the system that calls other catalogs. With both of these capabilities in place, the Data Catalog has transitioned from managing a directory of AWS tables to serving as an entry point to other catalogs.

3. Three Roles in the Catalog Layer

Three Roles in the Iceberg Catalog Layer on AWS
Three Roles in the Iceberg Catalog Layer on AWS
The developer guide for catalog federation describes this arrangement in a single paragraph. It is easy to read past, and it carries three separate roles.

Catalog federation uses AWS Glue Data Catalog to communicate with remote catalog systems to
discover tables and Lake Formation to authorize access to table data in Amazon S3.

Breaking it down, it consists of the following:

3.1 Resolving Where the Table Is

The first role is to determine the location of a table. Given a table name, it returns information such as where it resides in S3, the current schema, and the partitioning structure. The Data Catalog fulfills this role, and in a federated catalog configuration, it retrieves this information from a remote catalog.

What this role settles is that the authoritative metadata sits on the remote side. The Data Catalog does not possess the answers itself; it simply knows where to find them.

3.2 Reaching the Remote Catalog and Proving Who You Are

The second role is to establish connectivity and authentication. The remote catalog server sits outside AWS, so you need a path that reaches it over HTTP and a credential that says who you are. A separate object, the AWS Glue connection, holds that, and AWS Secrets Manager holds the credential itself.

What this role settles is who you are as the remote side sees it. The remote catalog doesn't recognize individual analysts; it only sees a single service principal. The set of objects that are visible on the remote side is therefore bounded by the permissions granted to that service principal.

3.3 Handing the Engine a Credential for the Data

The third role is vending a credential for the data itself. Because the engine reads the S3 objects directly, it has to hold a credential for S3. Lake Formation is what vends it. The catalog federation developer guide states:

Lake Formation manages access to table(s) by vending scoped credentials to the table data
stored in Amazon S3, allowing the engines to apply fine-grained permissions to federated
table(s).

What this role settles is whether the data can be read at all. That decision is made independently of the permissions on the remote side. The ability to see a table in the remote catalog and the ability to read the data within that table are two separate permissions.

3.4 The Three Are Configured Separately, and They Fail Separately

These three appear sequentially within a single console wizard, which can easily lead to the assumption that they represent a single configuration. However, in reality, they are independent. A state in which only two of the three are working comes about readily.

For example, if only the connection and authentication steps are successful, but the data access (credentials) fails, you'll be able to retrieve the table list, but SELECT statements will fail. Conversely, even if a table is deleted on the remote side, the permissions granted on this side will remain unchanged. The official documentation states both behaviors plainly, and Section 9 takes them one at a time.

The crucial initial assessment to make is this: do not judge whether catalog federation is set up from a single success or failure. Instead, evaluate each of the three roles individually, determining which components are currently succeeding and which are failing.

4. The Protocol Runs in Both Directions

The same Iceberg REST protocol runs in two directions, with the AWS Glue Data Catalog as the boundary. The two are easy to confuse, and when the direction changes, the side that holds the source of truth changes with it.

4.1 Outbound - The Data Catalog Calls a Remote Catalog

The outbound direction is catalog federation. You create a federated catalog inside the Data Catalog, and it mirrors a catalog object that lives on a remote catalog server. The console help panel describes what this mirroring does in a single sentence.

A federated catalog is an AWS Glue Data Catalog object that mirrors a remote catalog object in
a remote catalog server. It is used by AWS Glue to federate Iceberg REST API calls such as
ListDatabases, ListTables, GetTable, etc. to the remote catalog server.

In this direction, the authoritative metadata sits on the remote side. The AWS side only forwards the calls.

4.2 Inbound - The Data Catalog Answers Iceberg REST Clients

The inbound direction is the Data Catalog itself behaving as an Iceberg REST catalog. Endpoints sit under a Region-specific service endpoint; for example, in us-east-1, the endpoint would take the form:

https://glue.us-east-1.amazonaws.com/iceberg

The AWS Glue developer guide lists 12 Iceberg REST operations that this endpoint accepts. This list is accurate as of September 5, 2026.

CategoryOperation
ConfigurationGetConfig
NamespaceListNamespaces / CreateNamespace / LoadNamespaceMetadata / UpdateNamespaceProperties / DeleteNamespace
TableListTables / CreateTable / LoadTable / TableExists / UpdateTable / DeleteTable

This list should not be considered exhaustive. Future additions may cause discrepancies. The axis to judge by is what the specification has that this list does not. Of the 25 operations defined in the Apache Iceberg 1.6.1 OpenAPI specification, 13 are not included in the above list. These include 7 operations related to views (specifically, listViews, createView, loadView, replaceView, dropView, viewExists, and renameView), plus 6 additional operations: getToken, namespaceExists, registerTable, renameTable, reportMetrics, and commitTransaction. Knowing in advance which of these your client calls reduces how often you have to chase the operation list.

Authentication is AWS Signature Version 4. In this direction the gate is AWS IAM, not a token from the remote side. The AWS Glue developer guide states that reaching the endpoint requires the glue:GetCatalog IAM action. The same page writes it two ways: the prose has glue:getCatalog, while the operation table has glue:GetCatalog in four places. IAM does not distinguish the two when it evaluates a policy, so take the table's form.

In this direction, the authoritative metadata sits on the AWS side. External clients only come to read.

Another endpoint exists that provides operations not specified in the Iceberg REST specifications. Its path is /extensions, and it provides server-side scan plans. This is the pathway used when accessing tables within Amazon Redshift's managed storage, and it's a separate functionality from the federation with remote catalogs, which is the subject of this article.

4.3 Which Direction You Are In Decides Where the Source of Truth Sits

The two directions hold at the same time. In one account, one catalog can mirror a remote server while another is exposed outward. A design discussion therefore has to settle which direction it is about before anything else.

If you need to narrow the axis to one question, it is this: which side creates the table? The side that creates the table is the source of truth for its metadata. The side that only reads the table is dependent on the availability and permission model of the other side.

5. What a Federated Catalog Actually Does

5.1 Mirroring an Object, and Resolving at Query Time

One distinction here decides whether the rest of this article is correct. The federated catalog does not copy metadata.

The catalog federation developer guide carries two statements. The first is about synchronization.

Catalog federation synchronizes metadata across Data Catalog and remote catalogs when you access remote tables.

The second discusses the point of resolution.

When you query a federated table, Data Catalog discovers the latest table information in the
remote catalog at query time, getting the table's Amazon S3 location, current schema, and
partition information.

These two statements are not contradictory. They describe the same behavior using different phrasing. Synchronization occurs when you access remote tables; that is, at the time of access. There is no background job that periodically replicates metadata. The general availability announcement describes this as synchronizes metadata real-time.

Three consequences follow in practice:

  • Stale metadata never accumulates on this side. What comes back is what existed at the moment of the read.
  • Conversely, if the remote catalog is unavailable, queries on this side fail with it. There is no local copy to fall back on.
  • There is nothing to build for monitoring synchronization lag. In its place, the availability and response time of the remote catalog become preconditions for queries on this side.

The motivation for replacing a custom synchronization pipeline lies precisely in this point. What disappears is the synchronization process itself, not the dependency. What you depend on moves from the freshness of a copy to the availability of the remote catalog.

5.2 What Gets Forwarded, and What Those Calls Are Really Named

The AWS documentation names three of the calls it forwards. The sentence is the one from the help panel quoted earlier.

Iceberg REST API calls such as ListDatabases, ListTables, GetTable, etc.

Because such as and etc. are attached, this is not a declaration of the full set. It is an illustration. Reading it as a closed list leads to the wrong conclusion that only these three calls are forwarded.

Furthermore, two of these three names do not correspond to the operation names defined in the Iceberg REST specification. Among the 25 operationId values in the Apache Iceberg 1.6.1 OpenAPI definition, ListDatabases and GetTable are absent. The corresponding operations in the specification are listNamespaces and loadTable. Section 10 takes this up.

The practical implication for readers is as follows: When you need to verify the actual calls that are forwarded, consult the table of Iceberg REST operations in the AWS Glue developer guide. That table lists each operation alongside the corresponding IAM actions, Lake Formation permissions, and CloudTrail event names. For example, it states that the IAM action for LoadTable is glue:GetTable. You can find the correct mappings there.

5.3 One Level of Namespace on Both Sides

In a federated catalog configuration, both sides pin the namespace hierarchy to a single level.

The AWS Glue side of the constraint is stated on the Iceberg REST endpoint page.

Namespaces in Iceberg REST catalog APIs path can have multiple levels. However, AWS Glue only
supports single-level namespaces.

That constraint is about AWS Glue. A statement to the same effect covers the remote side too, but it is narrower. The page on federating to Snowflake states this, naming the Polaris catalog.

Nested namespaces in Polaris catalog are not supported. That is, catalog federation can access
remote Iceberg tables that follow a 3-part notation catalog.database.table.

That statement is about Polaris. The page on federating to Databricks carries no corresponding statement. How other remote catalogs treat nested namespaces is therefore something you check on each of their own pages. Because AWS Glue handles only a single level, though, the result fits the three-part form either way.

AWS Glue offers a workaround. You can name an intermediate catalog in the catalog path parameter, and the namespace below it then counts as a single level. However, this is a workaround for the Data Catalog's hierarchy and does not resolve any nested structures on the remote side.

One thing is worth checking before starting: into how many parts the dots split a table's fully qualified name in the remote catalog. If that count goes past three, the namespace design has to be revisited before federation begins.

6. Reaching the Remote Catalog

6.1 The Glue Connection Is a Separate Object With Its Own Lifecycle

The path to reach a remote catalog is created as a separate object, which is an AWS Glue connection. This connection is created using the aws glue create-connection command, specifying a ConnectionType appropriate for the type of remote catalog. An example for Snowflake, as shown in the developer guide, is:

aws glue create-connection \
    --connection-input '{
"Name": "your-glue-connection-to-snowflake-account",
"ConnectionType": "SNOWFLAKEICEBERGRESTCATALOG",
"ConnectionProperties": {
    "INSTANCE_URL": "your-snowflake-account-URL",
    "ROLE_ARN": "your-IAM-role-for-secrets-and-VPC-access",
    "CATALOG_CASING_FILTER": "LOWERCASE_ONLY"
},
"AuthenticationConfiguration": {
    "AuthenticationType": "OAUTH2",
    "OAuth2Properties": {
        "OAuth2GrantType": "CLIENT_CREDENTIALS",
        "TokenUrl": "your-internal-or-external-token-server-url",
        "OAuth2ClientApplication": {
            "UserManagedClientApplicationClientId": "our-client-id"
        },
        "TokenUrlParametersMap": {
            "scope": "all-apis"
        }
    },
    "SecretArn": "arn:aws:secretsmanager:your-aws-region:your-aws-account-id:secret:snowflake-secret"
}
}'

One object carries the URL it reaches, the IAM role it assumes, the casing filter for identifiers, and the authentication settings. This section covers the first three. The next section covers the authentication settings.

A connection and a federated catalog do not share a lifecycle. The developer guide states that one connection can back several federated catalogs, and that deleting a catalog does not delete the connection. If you want to delete a connection, you must first delete all related catalogs and then call aws glue delete-connection.

This asymmetry shows up in operations. When replacing credentials, you should modify the connection, not the catalog. Conversely, even if you remove a catalog, the underlying path to reach it remains. This results in two distinct units for inventory management.

6.2 Two Authentication Types, Two Reserved Key Names

AWS Glue connectors support two authentication methods. The developer guide describes them as follows:

The AWS Glue connector supports two authentication types - OAuth2 and Custom.

The key difference between the two methods lies in who is responsible for refreshing the tokens.

MethodInformation Stored in Secrets ManagerKey NameToken Refresh
OAuth2The client secretUSER_MANAGED_CLIENT_APPLICATION_CLIENT_SECRETAWS Glue automatically refreshes the token when it expires.
CustomThe access token itselfBEARER_TOKENThe customer's own application or system creates, refreshes, and manages the token.

The key names are reserved terms. The developer guide states the following regarding the OAuth2 method:

USER_MANAGED_CLIENT_APPLICATION_CLIENT_SECRET is a reserved keyword that AWS Glue uses to
refer to a client secret value in the secret. Use the same keyword when you are creating the
secret in Lake Formation console too.

The key name BEARER_TOKEN for the Custom method is documented in an article on the AWS Big Data Blog, rather than in the developer guide. The developer guide simply mentions storing access tokens but does not specify the key name. Check which source you are reading from before you configure it.

In the OAuth2 configuration, the client ID is not required. The page on federating to Snowflake states that omitting it makes the token request carry the secret alone.

The OAuth2ClientApplication field is optional. If you omit it, the token request authenticates
using only the client secret stored in AWS Secrets Manager, without sending a client ID.

The choice of method depends on who you want to manage the token's lifecycle. If you prefer AWS to handle the management, OAuth2 is the appropriate choice. If a token issuing system already exists and is meant to stay authoritative, Custom is the one. Choosing Custom, though, means an expired token surfaces as a failure on your side. That is one more thing to monitor.

6.3 The Casing Filter Decides Which Objects Exist at All

The connection property includes CATALOG_CASING_FILTER. This setting allows you to choose whether to ingest objects with lowercase identifiers or objects with uppercase identifiers from the remote catalog.

This setting is not about display. It decides what gets brought in at all. The page on federating to Snowflake puts it this way:

When you mount remote catalog with CATALOG_CASING_FILTER='UPPERCASE_ONLY' configuration,
databases and tables with uppercase identifiers are federated but objects with lowercase
identifiers are not.

In other words, the databases and tables whose identifiers are in the case you did not choose do not exist on this side. Nothing raises an error. They just never appear in the listing.

Problems arise when the remote catalog uses a mix of both conventions. How uppercase and lowercase identifiers are treated varies by product, and inside one catalog it can differ by when the object was created. Counting how the identifiers are distributed on the remote side is work that comes before federation.

6.4 Network Paths

The catalog federation developer guide names three shapes the path can take.

Catalog federation supports direct connections to remote catalog sources using standard HTTPS
connectivity. It also supports connectivity through Amazon VPC when you want to maintain
network isolation and connectivity using proxy support when you want secure communication
through organization firewalls.

When choosing a configuration that utilizes Amazon VPC, the IAM role used for the connection requires permissions to manage network interfaces. The example in the developer guide grants permissions for ec2:CreateNetworkInterface, ec2:DeleteNetworkInterface, and ec2:DescribeNetworkInterfaces, subject to VPC and subnet conditions.

One console capability differs. The page on federating to Databricks states that connection testing is unavailable when the connection goes through Amazon VPC.

Test connection functionality is not available when connecting to Databricks using Amazon VPC.

With one validation method removed, if you choose network isolation, you will need to provide an alternative means of verifying connectivity.

7. The Credential for the Data Is a Separate Decision

7.1 What Gets Registered With Lake Formation Is the Connection

This is the place where carrying over what you already know about Lake Formation will lead you wrong.

In standard Lake Formation, you register an S3 location. You use aws lakeformation register-resource and provide the ARN of the S3 bucket, associating it with an IAM role that grants access to that location. With a federated catalog, the object being registered is different: you provide the ARN of an AWS Glue connection.

aws lakeformation register-resource \
    --resource-arn your-glue-connector-arn \
    --role-arn your-IAM-role-ARN-having-LF-access \
    --with-federation \
    --with-privileged-access

This one call settles two things at once. First, it establishes that any tables appearing under this connection will fall under Lake Formation's governance. Second, it specifies the IAM role that Lake Formation will assume when reading data from those tables on S3.

The AWS Glue developer guide summarizes this outcome in a single sentence.

Federated catalogs in the Data Catalog have Lake Formation registered data locations.

There are two consequences of registering a connection. First, you no longer need to explicitly list the locations of remote tables on S3. Second, you must accurately estimate the scope of the buckets you grant access to via the IAM role. The example in the developer guide demonstrates an IAM policy that lists specific buckets. While AWS doesn't explicitly state what happens if remote tables are added and their data is placed outside the listed buckets, the structure of the policy suggests one outcome: metadata resolution will succeed through the connection, but data reads will be blocked by S3 permissions.

7.2 Two Roles Doing Two Different Jobs

A federated catalog configuration brings in two IAM roles of different character. When created through the console, both policies are assigned to a single role, making it appear as one. However, when created using the AWS CLI, they are separated.

RoleTrusted Service PrincipalPermissions Granted
Connection roleglue.amazonaws.comRetrieving, describing, and updating the Secrets Manager secret. If using an Amazon VPC, also includes operations on network interfaces.
Lake Formation rolelakeformation.amazonaws.coms3:GetObject and s3:ListBucket for the S3 buckets containing remote Iceberg tables. If encryption is enabled, also includes kms:Decrypt and kms:Encrypt for AWS KMS.

The page on federating to Snowflake notes how the console differs.

When you use Lake Formation console to create a federated catalog, the console uses a single
IAM role with both policies attached to complete setup.

The separation of these roles indicates two distinct trust boundaries. One handles credentials for the remote catalog, while the other interacts directly with the data. From an auditing perspective, maintaining this separation improves clarity. If you are deploying a configuration created through the console to a production environment, be aware that a single role is responsible for both functions.

7.3 What Scoped Means in Scoped Credentials

What does it mean, concretely, for a vended credential to be scoped?

One reference needs pinning down first. The catalog federation developer guide states in prose that Lake Formation vends scoped credentials, without naming the API that does it. In contrast, the Lake Formation developer guide defines a set of APIs that function as credential vending services, including GetTemporaryGlueTableCredentials. And as Section 7.1 quoted, a federated catalog in the Data Catalog has Lake Formation registered data locations.

What follows is the definition given for those credential vending APIs. AWS does not state outright that the federated catalog path calls that API itself. What this page does give is the definition of scoped in the Lake Formation sense. The API reference describes it as follows:

Allows a caller in a secure environment to assume a role with permission to access Amazon S3.
In order to vend such credentials, AWS Lake Formation assumes the role associated with a
registered location, for example an Amazon S3 bucket, with a scope down policy which restricts
the access to a single prefix.

So the content of scoped is the scope down policy, and the unit it narrows to is one prefix. Even if a registered role has permission to read an entire bucket, the vended credential reaches only that table's prefix.

What comes back is an ordinary set of temporary credentials. In addition to the AccessKeyId, SecretAccessKey, and SessionToken, it also returns a VendedS3Path and an Expiration date. This format is the same as that expected by the vended-credentials section of the Iceberg REST specification (as discussed in Section 2.2). An AWS service fills in the values, in a slot the specification left open.

The request can set the validity period. DurationSeconds is optional, and the same page gives two different maximums for it. Section 10 takes this up.

7.4 The Engine Declares What It Can Enforce

The vending request carries one more parameter that matters: SupportedPermissionTypes. With it, the caller declares which kinds of filtering the caller itself can enforce.

If the declaration does not line up with the permissions actually in place, the vending fails. The API reference defines this failure as a specific exception.

The engine does not support filtering data based on the enforced permissions. For example, if
you call the GetTemporaryGlueTableCredentials operation with SupportedPermissionType equal to
ColumnPermission, but cell-level permissions exist on the table, this exception is thrown.

This design means that the right to determine the granularity of permissions is not solely held by the catalog owner. Even if the catalog owner applies a cell-level filter, an engine that can enforce only column-level filtering does not receive the credentials. Nothing leaks. The table simply becomes unreadable from that engine.

Permission design therefore runs as a negotiation between two granularities: the one you want to grant, and the one the engine you want to let in can enforce. The table of which engine supports which granularity was already measured and published in Fine-Grained Access Control for AI Data with AWS Lake Formation. This article does not rebuild it. Instead, the purpose here is to illustrate how failures manifest when the values in that table do not align.

8. What the Engine Receives, and Where It Goes Next

Where Each Call Goes in a Single Federated Query
Where Each Call Goes in a Single Federated Query

8.1 The Catalog Returns a Location, Not the Data

A single query results in the call branching into two separate paths. The catalog federation developer guide states the branch outright.

Your analytics engine (Amazon Athena, Amazon Redshift, Amazon EMR) then uses this information
to access Iceberg data files directly from Amazon S3.

The catalog returns information about the S3 location, schema, and partition. It does not transmit the data itself. The engine receives this information and then directly accesses S3.

This branch is where the separation of the three roles becomes visible. The metadata path runs out to the remote catalog; the data path ends at S3. No credential is shared between them.

On the engines it supports, the catalog federation developer guide says this:

It is supported by a wide variety of analytics engines, including Amazon Redshift, Amazon EMR,
Amazon Athena, AWS Glue, third-party engines like Apache Spark, and more.

With including and and more attached, this is an illustration and not an exhaustive list. This article does not build an exhaustive table of supported engines. There are two reasons for this. First, cross-service lists of this kind tend to lag the individual product documentation. Second, the list is actually growing. AWS Clean Rooms, for one, announced support for remote Iceberg REST catalogs in February 2026, and it does not appear in the sentence above.

The axis to judge by is this. Does the product documentation for the engine you want to use carry a procedure for reading a federated catalog? If it does, the engine is supported. A name missing from the cross-service sentence above is not proof that it is unsupported.

8.2 Amazon Redshift Has Two Ways In, and They Have Different Constraints

The design of the access points varies for each engine. The Amazon Redshift database developer guide outlines two methods.

MethodReference FormatConstraints
Automount"awsdatacatalog"."database_name"."table_name" (three-part format). Schema creation is not required.The guide describes awsdatacatalog as covering the databases registered in the Data Catalog for the same account and Region, and separately recommends automount for cross-Region deployments. Both statements appear below.
External schemaPoints to the database of the resource link using CREATE EXTERNAL SCHEMA.Works only when the catalog, the compute, and the S3 bucket all sit in the same Region.

That guide marks the Region constraint on external schemas as Important.

External schema works only when all resources (catalog, compute, and Amazon S3 bucket) are in the same Region.

That guide recommends automount for cross-Region deployments as a best practice, because automount handles cross-Region resolution transparently and needs no schema management. The same page also describes automount as covering databases in the same account and Region. Both statements are recorded here as they stand. What is said without ambiguity is the condition on external schemas: the catalog, the compute, and the S3 bucket must all sit in the same Region.

In design order, fix the Region layout first and pick the access method second. If you proceed with designing permissions based on external schemas, you may end up needing to rebuild.

8.3 Writes Are Possible, and They Go Through the Same Permissions

A federated catalog is not read-only, and a write goes through the same Lake Formation permission model that a read goes through. The Lake Formation developer guide states that creating or deleting tables within a federated database requires the Lake Formation Create table and Drop permissions. It is also possible to execute DDL queries from Apache Spark on Amazon EMR.

That writes are possible and that writes are wise are two different statements. In a configuration where the authoritative metadata sits on the remote side, a write issued from the AWS side is still arbitrated, in the end, by the remote catalog. Settle which side owns that operational boundary before opening the path.

9. What This Design Takes Away

What catalog federation removes is the machinery that replicates metadata. Instead, certain assumptions are fixed. Everything listed here reflects behavior explicitly documented in the official documentation.

9.1 Metadata Can Federate Where Data Access Cannot

This is the one that breaks most quietly. On table formats other than Iceberg and object storage other than Amazon S3, the page on federating to Databricks states the following:

You can query Iceberg tables stored in Amazon S3 using this integration. When using any other
table format or object storage, you can federate metadata in remote catalogs to AWS Glue and
list its databases and tables but query operations like SELECT ColumnFoo from TableBar will
fail during query with error 'Failed to read Apache Iceberg table. Object storage location is
not supported.'

In other words, a table in the remote catalog that is not Iceberg still shows up in the listing on this side. It shows up, and it cannot be read. Stopping the check once the table listing comes back reads this state as normal.

The check to run is not the listing but a SELECT that returns a single row, against each table. If you want to access tables using the Delta format, you will need to ensure that Iceberg metadata is available on the remote side.

9.2 Dropping in the Remote Catalog Leaves the Grant Behind

Even when you drop a database or a table in the remote catalog, the permissions granted on this side through Lake Formation are not revoked automatically. The pages on federating to Snowflake and to Databricks both state this.

When you drop resources (like databases and tables) in the Databricks, Lake Formation does not
automatically revoke the permissions granted on that federated resource. To remove the access
permissions, you need to explicitly revoke the permissions that were previously granted on the
federated resource using Lake Formation.

The potential issue arises when tables are recreated with the same names. Existing grants will then apply to these new entities. The lifecycle of a table in the remote catalog and the lifecycle of a grant in Lake Formation do not move together. Ensure that your deletion procedures include revoking Lake Formation permissions.

9.3 The Casing Filter Silently Excludes Objects

Section 6.3 described the result of the CATALOG_CASING_FILTER. Objects whose identifiers are in the case that was not selected appear as absent rather than as an error. There is no way to distinguish between objects that do not exist and those that were filtered out.

9.4 A Ceiling on Metadata Size

The catalog federation developer guide names one limit for federated Iceberg catalogs.

Iceberg table metadata size limit – For federated Iceberg catalogs, AWS Glue Data Catalog
supports a maximum metadata size of 5 MB per REST API call. AWS Glue Data Catalog rejects
requests to tables with metadata exceeding this limit.

Metadata for Iceberg tables grows as more table snapshots accumulate, and as the history of partitions and schemas becomes longer. Therefore, this limit depends on whether the table is being actively maintained. Tables that are no longer maintained may eventually exceed this limit, rendering them inaccessible.

The design of table maintenance, specifically compaction and table snapshot expiration, is already covered in the existing AWS Data Lakehouse Architecture Guide. This article emphasizes that, given that the physical representation of a federated table resides remotely, its maintenance is the responsibility of the remote operations team. When approaching this limit, it's crucial to define who will take action as part of the federation design.

9.5 Nested Namespaces

As described in Section 5.3, namespaces are fixed to a single level on both sides. If the side that designed the remote catalog anticipated nested namespaces, the introduction of federation will necessitate changes to the design on the other side. This is a typical example of how technical constraints can lead to negotiations between organizations.

9.6 What the Documentation Does Not Say

This section records what could not be confirmed as well. The AWS documentation does not state how the AWS Glue Iceberg REST endpoint treats the X-Iceberg-Access-Delegation header. This is not mentioned in the list of 12 operations supported by the endpoint, nor on the individual pages describing each operation. As of the date of this article (September 5, 2026), this remains unverified.

Similarly, no statement could be found about the default value used when CATALOG_CASING_FILTER is omitted. State it explicitly when you configure the connection.

10. Where the Primary Sources Disagree

10.1 Why This Article Records Them

This section is included for two reasons. First, it ensures that readers encountering the same materials will recognize the same inconsistencies. Second, it highlights that the appropriate course of action depends on the type of inconsistency. In cases where prose and machine-readable definitions conflict, prioritize the latter. When discrepancies arise between different documents, prioritize the wording of the party imposing the constraints.

The information presented here is current as of September 5, 2026.

10.2 One API Page Gives Two Maximum Durations

The API reference for GetTemporaryGlueTableCredentials specifies an upper limit for DurationSeconds in two different places, with differing values. The descriptive text states:

The time period, between 900 and 21,600 seconds, for the timeout of the temporary credentials.

The machine-readable range definition, located directly beneath the same section, states:

Valid Range: Minimum value of 900. Maximum value of 43200.

While the lower limit is consistent at 900, the upper limit differs by a factor of two, being 21,600 and 43,200. The machine-readable Valid Range is the one to take. The input validation on the API is the more likely to follow it. However, from a design perspective, it is safer to keep the value at or below the 21,600 the prose gives, since that passes under either reading.

10.3 The Same Page Counts the Permission Types Twice, and Gets Two Answers

On the same page, the SupportedPermissionTypes section lists permission types, but there's a discrepancy between the descriptive text and the enumerated list. The descriptive text mentions two types.

A list of supported permission types for the table. Valid values are COLUMN_PERMISSION and CELL_FILTER_PERMISSION.

The enumerated list directly below lists four types.

Valid Values: COLUMN_PERMISSION | CELL_FILTER_PERMISSION | NESTED_PERMISSION | NESTED_CELL_PERMISSION

The descriptive text omits NESTED_PERMISSION and NESTED_CELL_PERMISSION. The names read as permissions over columns that carry a nested structure, but the page says nothing more about them.

This omission has practical consequences. As Section 7.4 showed, if the declared values do not match the permissions actually in place, the vending fails. Reading the prose alone and concluding that only two values can be declared drops two supported values from the candidate list. The API enumeration should be considered the definitive source.

10.4 Two of the Three Named Calls Are Not in the Iceberg REST Specification

As mentioned in Section 5.2, there is a discrepancy. Both the Lake Formation console's help panel and the AWS Big Data Blog describe the calls a federated catalog forwards as Iceberg REST API calls, citing ListDatabases, ListTables, and GetTable.

However, out of the 25 operationId values included in the Apache Iceberg 1.6.1 OpenAPI definition, ListDatabases and GetTable are not present. The corresponding operations in the specification are listNamespaces and loadTable. Only ListTables appears in both.

What these two names resemble becomes clear from the table of Iceberg REST operations in the AWS Glue developer guide. That table lists the IAM actions associated with each operation; the IAM action for ListNamespaces is glue:GetDatabase, and the IAM action for LoadTable is glue:GetTable. The names mentioned by the help panel and the blog are closer to AWS Glue's terminology rather than the operation names defined in the protocol. However, they are not exact matches. The API in AWS Glue that lists databases is GetDatabases, and as of this article's verification date, no API named ListDatabases could be found in the AWS Glue API reference.

Therefore, when discussing the protocol, it is best to use the names defined in the Iceberg specification. When discussing IAM policies and CloudTrail, use the IAM action names used by AWS Glue. There are existing documents that mix these two terminologies, so it is important not to rely solely on the names to determine which layer is being discussed.

10.5 Two Adjacent Operations State Opposite Rules About purge

The Iceberg REST operations table in the AWS Glue developer guide contains two operations related to table deletion that describe opposite actions regarding purgeRequest.

The considerations for DeleteTable are as follows:

purgeRequest=true is not supported.

The considerations for the adjacent operation, StartDeleteTableTransaction, are as follows:

purgeRequest=false is not supported.

Although these are separate operations, they are logically compatible. One reading is that the synchronous delete cannot purge the data, while the asynchronous transaction cannot decline to purge it. However, the same section regarding DeleteTable also states that the operation will fail if purge = TRUE is specified for tables on S3, while it will fail if purge = FALSE is specified for tables on Amazon Redshift managed storage. There is inconsistency in the use of parameter names, alternating between purge and purgeRequest, and the conditions are not consistently described.

The practical conclusion is this: do not rely solely on this table to determine whether deleting a table will also delete the data files. Instead, verify this behavior in a testing environment.

10.6 Federated Catalog Names Two Different AWS Features

The last one is not a contradiction but a collision. The term "federated catalog" in AWS documentation refers to two distinct features.

One is the federation with remote Iceberg catalogs, which is the subject of this article. The other is registering Athena federated query connectors as catalogs inside the Data Catalog. Regarding the latter, the Athena user guide lists the connectors it accepts and gives the following constraints.

DDL operations are not supported on federated catalogs.

This passage does not apply to the subject of this article. As Section 8.3 showed, federation with remote Iceberg catalogs supports DDL. When searching for the same term, users may encounter both descriptions, which can give the impression that one contradicts the other.

The key to distinguishing between them is whether the context mentions a remote catalog server. If it does not, and instead lists JDBC data source names, it is about Athena federated query.

11. Failure Modes

The types raised so far are reordered here from the symptom side. None of them surfaces as a clear error.

SymptomWhat's likely happeningWhere to check
Able to list tables, but SELECT failsOnly the metadata path is accessible; the data path is not. The table format is not Iceberg, or the storage is not Amazon S3, or the S3 range covered by the IAM role registered with Lake Formation does not include that table.IAM role permissions for the bucket used in the connection. Remote table format.
Tables that should be listed are not visibleCATALOG_CASING_FILTER is set to the opposite case. Alternatively, the service principal on the remote side may lack the necessary permissions to read the metadata for that object.Connection properties. Permissions for the service principal on the remote side.
Permissions remain on tables that should have been deletedDeletion on the remote side isn't invalidating the permissions granted through Lake Formation.List of Lake Formation grants.
Credential vending fails from one engine onlyThe table carries permissions finer than the SupportedPermissionTypes the engine declared. A PermissionTypeMismatchException comes back.Granularity of filters applied to the table. Granularity supported by the engine.
Tables that were previously accessible are now inaccessibleThe table's metadata exceeds the 5 MB limit per REST API call.Status of table maintenance on the remote side.
Queries fail even after creating an external schemaOne of the three sits in a different Region: the catalog, the compute, or the S3 bucket.Where the three sit. Whether automount is an option.
A table whose name has four parts cannot be federatedBoth sides are enforcing a single-level namespace.Namespace hierarchy on the remote side.

One property runs through the whole list. None of these shows up right after the setup. What gets checked right after the setup is usually the table listing, and at that moment only the first of the three roles has been exercised.

To ensure proper setup, there are three essential steps to follow. First, obtain a list of all tables. Second, run a SELECT that returns a single row from each engine that is meant to read the table. Finally, with the most granular permissions you intend to grant already applied, run that same SELECT statement again. Only after completing all three steps can you confirm that all three roles have been properly verified.

12. Frequently Asked Questions

Does creating a federated catalog copy metadata to AWS?

No. The catalog federation developer guide states that the latest table information is discovered in the remote catalog at query time. There is no background process that periodically copies metadata. Therefore, a situation where outdated metadata remains on this side will not occur. Instead, if the remote catalog fails to respond, queries on this side will also fail.

While the remote catalog is unavailable, can queries keep running from a cache?

No, for the same reason as the previous answer. Metadata resolution occurs at runtime, so the availability of the remote catalog is a prerequisite for queries on this side. This dependency is the first trade-off to weigh when replacing a synchronization pipeline you built yourself.

Are federated catalogs and AWS Glue's Iceberg REST endpoints the same thing?

No. They operate in opposite directions. A federated catalog has the Data Catalog calling a remote catalog; the Iceberg REST endpoint has the Data Catalog answering external clients. While both use the same Iceberg REST protocol, the direction of metadata ownership is reversed.

If a table is deleted in the remote catalog, will the permissions on the AWS side also be removed?

No. Lake Formation does not automatically revoke permissions granted to federated resources. You need to explicitly revoke them. If a table is recreated with the same name, the permissions left behind apply to the new entity.

What happens with tables that are not Iceberg?

They will appear in the listing, but queries will fail. The page on federating to Databricks states that for table formats other than Iceberg and object storage other than Amazon S3, the databases and tables can be listed but the query fails at query time. If you want to read Delta format tables, ensure that Iceberg metadata is available on the remote side.

Which engines support federated catalogs?

The catalog federation developer guide gives Amazon Redshift, Amazon EMR, Amazon Athena, AWS Glue, and Apache Spark as examples, and that is not an exhaustive list. The set is in fact growing. AWS Clean Rooms announced support in February 2026. Determine compatibility by checking the product documentation for the engine you want to use. A name missing from a cross-service list is not proof that the engine is unsupported.

In a federated catalog, is what you register with Lake Formation an S3 location?

No. In a federated catalog configuration, what you register is the ARN of the AWS Glue connection. The key difference from a standard Lake Formation setup lies here. As a result, you no longer need to enumerate the exact location of remote tables within S3, but you will need to accurately estimate the scope of buckets that the registered role can access.

How far does a vended credential reach?

To a single prefix. The API reference states that Lake Formation assumes the registered role and applies a scope down policy that restricts access to a single prefix. Even if the registered role can read the entire bucket, the vended credential reaches less than that.

If permissions finer than column level are applied, does the table become unreadable from every engine?

No. Engines that can enforce that granularity still read the table. Only the vending request from an engine that cannot enforce it fails, with a PermissionTypeMismatchException. A table detailing which engines support which granularities is available in the published Fine-Grained Access Control for AI Data with AWS Lake Formation.

Can you directly federate catalogs with two or more levels of remote namespaces?

No. AWS Glue handles only single-level namespaces. On the remote side, the page on federating to Snowflake states that catalog federation reaches tables that follow the three-part notation. Before starting, count into how many parts the dots split a table's fully qualified name on the remote side.

Is it possible to write using this configuration?

Yes. Creating or deleting a table in a federated database needs the Lake Formation Create table and Drop permissions. You can also execute DDL queries from Apache Spark on Amazon EMR. However, writing from the AWS side while the authoritative metadata sits on the remote side leaves the operational boundary unclear. Settle that boundary before you open the path.

13. Summary

The catalog layer, while appearing as a single layer, is composed of three independent decisions.

The first decision concerns which side holds the authoritative metadata for the table. In a federated catalog configuration, the authoritative metadata sits on the remote side. AWS only retrieves the metadata when a query is executed; it does not maintain a local copy. This choice eliminates the need for ongoing synchronization, instead making the availability of the remote catalog a prerequisite for query execution.

The second decision involves determining the path to reach the remote catalog and establishing the identity for accessing it. A separate object, the AWS Glue connection, holds it, and Secrets Manager holds the credential. This path does not share a lifecycle with the federated catalog. One connection can back several catalogs, and deleting a catalog leaves the connection in place.

The third decision concerns which party vends the credential for the data on S3. On AWS that party is Lake Formation. What gets registered with it is an AWS Glue connection, not the S3 location that a standard Lake Formation setup would register. The credential it vends reaches a single prefix. And the finer the permissions you grant, the more the reading engine's own capability becomes a precondition.

Three decisions settled separately are three decisions that fail separately. Being able to list a table but not read it, and finding a permission still attached to a table that was deleted, are both behaviors the official documentation states plainly. Stopping the check once the table listing comes back is what makes these states easy to miss.

The same Iceberg REST protocol appears once more, with its direction reversed. The AWS Glue Iceberg REST endpoint publishes the Data Catalog as a REST catalog for external clients. When discussing catalog federation, the first thing to definitively establish is whether the current discussion refers to one direction or the other. Settle the direction and you settle which side is authoritative. Settle which side is authoritative and the availability dependency and the permission dependency follow.

14. References



References:
Tech Blog with curated related content

Written by Hidekazu Konishi