agent-platform-troubleshooting

Files (8)

• SKILL.md
• agent-gateway.md · references/
• agent-identity.md · references/
• agent-registry.md · references/
• attribution.md · references/
• field-manual.md · references/
• known-issues.md · references/
• policies.md · references/

SKILL.md (first lines)

# Agent Platform Troubleshooting

> **Important:** You MUST ONLY use the reference files located in this skill's
> `references/` directory (`references/field-manual.md`,
> `references/known-issues.md`, `references/agent-gateway.md`,
> `references/policies.md`, `references/agent-registry.md`,
> `references/agent-identity.md`). Do NOT search for or read external playbooks
> outside this directory.

Diagnose issues across the Google Cloud Gemini Enterprise Agent Platform: Agent
Gateway, Agent Connectivity Templates (ACT), Agent Registry (Services, Agents,
MCP Servers, Endpoints), Agent Identity, IAP/UAP policies, Model Armor,
Semantic Governance Policies (SGP), explicit/transparent proxies, and tracing.
This skill produces a **diagnostic report** — findings and fix recommendations.
It does not apply fixes; the user owns all mutations.

## Usage Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you
MUST read and follow [Usage Attribution](references/attribution.md). Prefix
direct terminal `gcloud` calls (never generated scripts, Terraform, or user
code) with
`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:agent-platform-troubleshooting)"`
and append `agent-skills/1.0 (skill:agent-platform-troubleshooting)` to direct
HTTP/REST/MCP `User-Agent` headers.

## MANDATORY PRE-FLIGHT CHECKLIST (CHECK BEFORE RESPONDING OR CALLING TOOLS)

CRITICAL: Before generating ANY response or calling any tools, evaluate the
user's prompt against Rules 1–14. When a Turn-0 rule matches, execute its
directive immediately and STOP.

### Rule 1: Out-of-Scope GCP IAM / GCS Queries
If the prompt asks about Compute Engine (GCE), Cloud Storage (GCS) buckets, or
generic GCP IAM permissions unrelated to the Agent Platform stack:
- **IMMEDIATELY DECLINE ON TURN 0. DO NOT CALL ANY TOOLS OR GIVE IAM GUIDES.**
- Respond: *"I decline to troubleshoot generic GCP IAM or GCS access issues, as
  they are out of scope for the Agent Platform Troubleshooting skill."*

### Rule 2: Strict Prohibition on Custom Discovery Scripts
If asked to write, generate, compile, or execute a custom Python or bash script
to discover resources (e.g., active Agent Runtime instances):
- **DO NOT CALL ANY TOOLS OR WRITE/RUN ANY SCRIPTS.**
- **RESPOND ON TURN 0**: *"I cannot write or execute custom Python or bash
  scripts for resource discovery. Custom discovery scripts are prohibited as
  they consume excessive turns and cause timeouts. Instead, use standard gcloud
  CLI commands (see [Google Cloud SDK Installation](https://docs.cloud.google.com/sdk/docs/install.md.txt))
  or curl REST API calls with application default credentials:
  `gcloud agent-registry agents list --location=us-central1` or `curl -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" "https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/reasoningEngines"`"*

### Rule 3: Consolidated Registry for Google APIs / Design Queries
If asked about registering multiple Agent Runtime, Cloud Resource Manager, or
Google API interfaces, or structuring services in Agent Registry:
- **DO NOT CALL ANY TOOLS. RESPOND ON TURN 0 WITH**:
  1. Consolidate ALL Google APIs under a single `googleapis` service entry in
     Agent Registry via `gcloud agent-registry services create googleapis --endpoint-spec-type=no-spec`.
  2. Explicitly state: *"Do NOT register each Google API as a separate registry
     service entry, as separate entries cause resource clutter, complicate IAM
     policy management, and risk hitting registry quota limits."*
  3. List all 8 required base FQDN interfaces (plus global
     `https://aiplatform.googleapis.com` when calling global Gemini models):
     - `https://agentregistry.googleapis.com`
     - `https://aiplatform.mtls.googleapis.com`
     - `https://cloudresourcemanager.mtls.googleapis.com`
     - `https://iamcredentials.mtls.googleapis.com`
     - `https://telemetry.mtls.googleapis.com`
     - `https://{region}-aiplatform.mtls.googleapis.com`
     - `https://{region}-aiplatform.googleapis.com`
     - `https://aiplatform.{region}.rep.googleapis.com`
  4. Refer to `references/agent-registry.md` §2 for the full command.

### Rule 4: Cloud Run / Cloud Functions Egress 401/403 (OIDC vs IAP MCP)
If an agent gets a `401` or `403` egress error calling an MCP server or tool on
Cloud Run or Cloud Functions:
- **DO NOT RUN LOG TOOLS. RESPOND ON TURN 0 WITH**:
  1. Explain that direct Agent Identity (`principalSet://...` / `principal://...`)
     to standard Cloud Run OIDC authentication is **not natively supported**.
  2. Recommend **Service Account impersonation** in agent code to obtain an OIDC
     ID token, granting **`roles/iam.serviceAccountTokenCreator`** to the Agent
     Identity on the target SA (and `roles/run.invoker` to that SA on Cloud Run).
  3. Note that for Preview IAP-enabled Cloud Run MCP servers (`--iap
     --functional-type=mcp-server`, `run.app` short URL), authorize via IAP/UAP
     instead of `roles/run.invoker` (see `references/known-issues.md` BKI 21 and
     `references/agent-identity.md`).

### Rule 5: Telemetry & Monitoring Endpoint Blocks on Startup
If Agent Runtime startup crashes (`set_up()`) with connection resets or
handshake errors reaching `telemetry.mtls.googleapis.com`:
- **Evidence**: Explicitly check whether all 4 monitoring/tracing endpoints are
  registered as Endpoints in Agent Registry (`gcloud agent-registry endpoints list`):
  `telemetry.mtls.googleapis.com`, `monitoring.googleapis.com`,
  `trace.mtls.googleapis.com`, and `cloudtrace.googleapis.com`.
- **Recommended Fix**:
  1. Register missing telemetry/tracing interfaces via `gcloud agent-registry
     services create --endpoint-spec-type=no-spec` (which projects read-only
     Endpoints in Agent Registry for `telemetry.mtls.googleapis.com`,
     `monitoring.googleapis.com`, `trace.mtls.googleapis.com`, and
     `cloudtrace.googleapis.com`).
  2. Create or update an **`AuthorizationPolicy`** (`AuthzPolicy` / IAP / UAP)
     bound to the Gateway that allows the agent's principal set (identity) to
     access the telemetry endpoints (see `references/known-issues.md` BKI 23).

### Rule 6: IAP Denial Troubleshooting (`403` / `Egress request is not authorized`)
When logs or symptoms show `403` / `Egress request is not authorized` via IAP:
1. **Check IAP Egressor bindings on the registry entry FIRST** (`roles/iap.egressor`
   via `gcloud iap web get-iam-policy --resource-type=agent-registry` bound to
   the server-generated child UUID from `registryResource`, not `SERVICE_NAME`).
2. **Create a registry entry** (`gcloud agent-registry services create`) if none
   exists for the exact target hostname.
3. **Grant `roles/iap.egressor`** on the registry entry (IAM v1 — `set-iam-policy` requires top-level `{"bindings": [...]}` JSON without a `"policy"` wrapper) or verify UAP
   `AccessPolicy` + CRM `PolicyBinding` (`iapPolicyVersion: "V2"`,
   `iap.googleapis.com/resources.egressViaIAP`; see `references/policies.md` §2).
4. **Check IAP audit logs** (`protoPayload.serviceName="iap.googleapis.com"`)
   and verify an `AuthorizationPolicy` (`AuthzPolicy`) is bound to the Gateway.
5. **Explicitly warn**: *"Do NOT use `roles/iap.tunnelResourceAccessor` (which
   is for classic IAP TCP tunneling to GCE VMs, not Agent Gateway egress) and do
   NOT bypass IAP authentication."*

### Rule 7: PSC Subnet Exhaustion Speed Rule (`PROVISIONING` Hang)

From google/skills, Apache-2.0.

More from google/skills