federate-lakehouse-catalog

Files (1)

• SKILL.md

SKILL.md (first lines)

# Federate Lakehouse Catalog via Cross-cloud Lakehouse

This skill describes how to set up a federated catalog in BigQuery to query
remote catalogs like Databricks Unity Catalog or AWS Glue Data Catalog data in
AWS over the public internet.

## Prerequisites

-   For Databricks: Databricks Workspace URL and OAuth Service Principal (Client
    ID and Secret) with read access.
-   For AWS Glue: AWS Administrator access to create IAM roles and permissions
    policies.
-   Active Google Cloud project with administrative access to create lakehouse
    resources, and secrets in the case of Databricks.

## Procedure

### Step 1: Information Gathering and Region Selection

Before running any commands, the agent **MUST** collect the following
information from the user:

1.  Determine which catalog the user wants to federate to (e.g., Databricks
    Unity or AWS Glue) and verify it is supported.
2.  Determine where the remote data is located (the specific AWS region).
3.  Using the Region Pairing Best Practice in the Gotchas section, help the user
    pick the optimal GCP region to minimize latency.
4.  Collect the necessary configuration variables for the chosen flow (e.g.,
    Databricks credentials or AWS Account ID).

Only proceed to the next steps once this information is confirmed.

### Step 2: API Verification

Verify that the required Google Cloud APIs are enabled for the project:

```bash
gcloud services check biglake.googleapis.com
```

If the API is not enabled, explicitly ask the user for permission to enable it.
Do NOT proceed without their confirmation.

### Flow A: Databricks Unity Catalog

#### 1. Create a Regional Secret for Credentials

Store the Databricks client ID and secret in Secret Manager. Ensure the
`secretmanager.googleapis.com` API is enabled. The secret **MUST** be in the
same region as your Lakehouse catalog.

1.  Create a JSON file named `credentials.json`:

```json
{
  "client_id": "<CLIENT_ID>",
  "client_secret": "<CLIENT_SECRET>"
}
```

1.  Set the Secret Manager API endpoint override for the region:

```bash
gcloud config set api_endpoint_overrides/secretmanager https://secretmanager.<REGION>.rep.googleapis.com/
```

1.  Create the secret:

```bash
gcloud secrets create <SECRET_NAME> \
  --location="<REGION>" \
  --project="<PROJECT_ID>" \
  --data-file=credentials.json
```

#### 2. Create the Federated Catalog

Create a BigLake Iceberg catalog of type `federated` pointing to Databricks.

```bash
gcloud alpha biglake iceberg catalogs create <CATALOG_NAME> \
   --project="<PROJECT_ID>" \
   --primary-location="<REGION>" \
   --catalog-type="federated" \
   --federated-catalog-type="unity" \
   --secret-name="projects/<PROJECT_ID>/locations/<REGION>/secrets/<SECRET_NAME>" \
   --unity-instance-name="<UNITY_INSTANCE_NAME>" \
   --unity-catalog-name="<UNITY_CATALOG_NAME>" \
   --refresh-interval="300s"
```

#### 3. Grant Catalog Access to the Secret

Grant the service account created for the catalog access to read the secret.

1.  Get the service account email by describing the catalog:

```bash
gcloud alpha biglake iceberg catalogs describe <CATALOG_NAME> \
    --project="<PROJECT_ID>" \
    --location="<REGION>" \
    --format="value(biglake-service-account-id)"
```

1.  Grant access:

```bash
gcloud secrets add-iam-policy-binding <SECRET_NAME> \
  --project="<PROJECT_ID>" \
  --location="<REGION>" \
  --member="serviceAccount:<SERVICE_ACCOUNT_EMAIL>" \
  --role="roles/secretmanager.secretAccessor"
```

### Flow B: AWS Glue

#### 1. Create the AWS IAM role with a placeholder trust policy

Lakehouse provisions a Google service account ID after catalog creation. Create

From jerrylin96/dotgemini, Apache-2.0.

More from jerrylin96/dotgemini