mlops••4 min read

Verifying GCP Workload Identity Federation (WIF) from CI/CD

Why long-lived service account JSON keys are a major liability and how to test and verify GCP Workload Identity Federation token exchange cleanly.

In modern enterprise DevOps, storing static service account JSON keys in GitLab CI variables or GitHub Actions secrets is an anti-pattern. If a key is leaked, copied into job logs, or stored indefinitely without rotation, it represents a severe security vulnerability.

Google Cloud's Workload Identity Federation (WIF) eliminates static credentials entirely. It allows external workloads (GitLab, GitHub, Kubernetes clusters) to authenticate to GCP by exchanging an ephemeral OIDC (OpenID Connect) token for short-lived Google Cloud access tokens.

However, when configuring WIF for the first time, debugging a failed token exchange can be frustrating. Pipelines fail with opaque 403 Forbidden or invalid_grant errors.

Here is a straightforward runbook for validating your WIF setup cleanly using step-by-step CLI verification.


1. How the WIF Flow Works

Before debugging, understand the three-step token exchange:

text
[CI Runner]
     │  1. Generates short-lived OIDC JWT (e.g., CI_JOB_JWT_V2 in GitLab or ACTIONS_ID_TOKEN_REQUEST in GitHub)
     ▼
[GCP Security Token Service (STS)]
     │  2. Validates JWT signature, issuer, and claims against Workload Identity Pool Provider
     ▼
[GCP Service Account Impersonation]
     │  3. Exchanges STS token for short-lived GCP Service Account OAuth2 access token
     ▼
[GCP Resources (GKE, Cloud Storage, Artifact Registry)]

If any single link in this chain has an attribute mismatch, the exchange fails.


2. Inspecting the OIDC Token First

The most common reason for failure is an attribute condition or attribute mapping mismatch (e.g., expecting assertion.sub to match a specific repository path or branch).

Inside your CI pipeline, output the raw decoded payload of your runner's OIDC token.

In GitLab CI:

yaml
test_oidc_token:
  stage: test
  id_tokens:
    MY_OIDC_TOKEN:
      aud: "https://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/MY_POOL/providers/MY_PROVIDER"
  script:
    # Print JWT payload (middle section between dots)
    - echo "$MY_OIDC_TOKEN" | cut -d'.' -f2 | base64 -d 2>/dev/null || true

In GitHub Actions:

yaml
jobs:
  inspect-token:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
    steps:
      - name: Inspect Claims
        run: |
          echo "Audience and issuer claims should match GCP pool provider config."

Inspect the decoded JSON:

  • Check iss (Issuer URL): Does it match the issuer configured on your GCP Pool Provider?
  • Check aud (Audience): Does it match the full resource name of your provider?
  • Check sub or project_path: Are repository names, usernames, or branch references matching the exact case and spelling configured in your IAM bindings?

3. Manually Testing Token Exchange via cURL

You do not need to push 20 dummy commits to test your pipeline. You can capture the OIDC token from your runner and perform the two-step exchange via curl to identify the exact error response.

Step 1: Exchange OIDC Token with GCP STS

bash
OIDC_TOKEN="<your_captured_jwt_token>"
PROVIDER_RESOURCE="//iam.googleapis.com/projects/<PROJECT_NUM>/locations/global/workloadIdentityPools/<POOL_NAME>/providers/<PROVIDER_NAME>"

# Request federated STS token
STS_RESPONSE=$(curl -s -X POST https://sts.googleapis.com/v1/token \
  -H "Content-Type: application/json" \
  -d "{
    \"audience\": \"$PROVIDER_RESOURCE\",
    \"grantType\": \"urn:ietf:params:oauth:grant-type:token-exchange\",
    \"requestedTokenType\": \"urn:ietf:params:oauth:token-type:access_token\",
    \"scope\": \"https://www.googleapis.com/auth/cloud-platform\",
    \"subjectTokenType\": \"urn:ietf:params:oauth:token-type:id_token\",
    \"subjectToken\": \"$OIDC_TOKEN\"
  }")

echo "$STS_RESPONSE"

If this step fails:

  • 400 Bad Request / invalid_grant: The issuer URL, audience, or token expiration is rejected by the provider.
  • 403 Forbidden: Your attribute condition on the provider evaluated to false (e.g., the branch or ref was disallowed).

Step 2: Exchange STS Token for Service Account Access Token

If Step 1 succeeds, extract the access_token from $STS_RESPONSE and impersonate the target Google Service Account:

bash
FEDERATED_TOKEN=$(echo "$STS_RESPONSE" | jq -r '.access_token')
SA_EMAIL="ci-deployer@your-gcp-project.iam.gserviceaccount.com"

SA_RESPONSE=$(curl -s -X POST \
  "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/${SA_EMAIL}:generateAccessToken" \
  -H "Authorization: Bearer $FEDERATED_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "scope": ["https://www.googleapis.com/auth/cloud-platform"],
    "lifetime": "3600s"
  }')

echo "$SA_RESPONSE"

If this step fails:

  • 403 Permission Denied: The target Service Account is missing the roles/iam.workloadIdentityUser binding. The member binding must match your federated identity:
    text
    principalSet://iam.googleapis.com/projects/<PROJECT_NUM>/locations/global/workloadIdentityPools/<POOL_NAME>/attribute.repository/<REPO_NAME>

4. Testing GCP Access with the Token

Once Step 2 returns an access token (ya29...), test an actual GCP API call:

bash
ACCESS_TOKEN=$(echo "$SA_RESPONSE" | jq -r '.accessToken')

# Test listing storage buckets or GKE clusters
curl -s -H "Authorization: Bearer $ACCESS_TOKEN" \
  "https://storage.googleapis.com/storage/v1/b?project=your-gcp-project"

If this returns your bucket list, your federation pipeline is 100% verified.


Summary

WIF issues are almost never random; they stem from:

  1. Mismatched aud / provider resource names.
  2. Inaccurate attribute condition filters (regex or string matching).
  3. Missing roles/iam.workloadIdentityUser permission on the destination Service Account.

Deconstructing the exchange into raw cURL steps isolates whether the failure is at the STS provider layer or the Service Account impersonation layer in seconds.

Technical Discussion (0)

Ask a question or reply to a thread below. Your email is never displayed publicly.

No comments yet. Start the technical discussion above.