Why Your GCP Service Account Alone Can’t Decrypt with CMEK (and How It Differs from AWS)

Introduction:

When I first moved from AWS to GCP, I expected things to work the same. In AWS, if my IAM role has kms:Encrypt and kms:Decrypt, I can upload and download S3 objects encrypted with KMS (SSE-KMS).

So when I set up my GKE cluster in GCP, I created a service account, gave it KMS permissions, and assumed I was done. But downloads kept failing with permission errors.

After troubleshooting, I realized that GCP works differently. Instead of your service account talking to KMS directly, GCP uses a service agent — a Google-managed account that belongs to the service itself. That’s the key difference, and once I understood it, everything started to make sense.

Service Accounts vs. Service Agents in GCP

Let’s clear up the terms first:

• User-managed service accounts → Created and controlled by you. These are what you attach to workloads (VMs, clusters, etc.).
• Service agents → Google-managed accounts that exist automatically in your project whenever you use certain services. Each is unique per project and tied to one service. They represent the service backend (for example Cloud Storage, Pub/Sub, Secret Manager, BigQuery, and others) whenever that service needs to perform actions on resources like Cloud KMS keys.
  
  These identities are not created manually by you, but are provisioned by Google so that the service itself can authenticate and act on your behalf when carrying out encryption, replication, or other internal operations.

In AWS terms, you can think of service agents as if S3 or Secrets Manager had their own IAM role that always contacts KMS on your behalf.
How Cloud Storage + CMEK Works

Upload (Write) Flow

1. Caller → Cloud Storage: Your app’s service account calls storage.objects.create.
2. IAM Check: Cloud Storage checks that the caller has storage.objects.create.
3. DEK Generation: Cloud Storage generates a Data Encryption Key (DEK).
4. Service Agent → KMS: The Cloud Storage service agent calls Cloud KMS to wrap the DEK with your CMEK.
5. Wrapped DEK Returned: KMS encrypts the DEK and sends it back.
6. Persist: Cloud Storage stores the encrypted object + wrapped DEK.

Download (Read) Flow

1. Caller → Cloud Storage: Your app’s service account calls storage.objects.get.
2. IAM Check: Cloud Storage checks that the caller has storage.objects.get.
3. Service Agent → KMS: The service agent unwraps the DEK with KMS.
4. Decryption: Cloud Storage uses the DEK to decrypt the object.
5. Response: Plaintext object is returned to the caller.

Key point: Giving your own service account KMS permissions won’t help. Only the Cloud Storage service agent is allowed to call KMS.
Why GCP Uses Service Agents

At first, this design feels unusual if you’re coming from AWS. But here’s why GCP uses it:

Separation of responsibilities

• Caller permissions control what you can do with the service.
• Service agent permissions control what the service can do with KMS.

Background operations

Services often need KMS access even when you’re not directly calling them — for example, during replication, key rotation, or internal migrations. Your app’s service account can’t handle those operations, but the service agent can.

Consistency

Every service that supports CMEK has its own predictable service agent identity:

service-<PROJECT_NUMBER>@gcp-sa-<SERVICE>.iam.gserviceaccount.com

CMEK with Other Services

Secret Manager

• Service Agent:
  service-<PROJECT_NUMBER>@gcp-sa-secretmanager.iam.gserviceaccount.com
• Needs: roles/cloudkms.cryptoKeyEncrypterDecrypter on the KMS key.
• Caller needs only roles/secretmanager.secretAccessor.

Pub/Sub

• Service Agent:
  service-<PROJECT_NUMBER>@gcp-sa-pubsub.iam.gserviceaccount.com
• Needs: roles/cloudkms.cryptoKeyEncrypterDecrypter on the KMS key.
• Caller needs only roles/pubsub.publisher or roles/pubsub.subscriber.

The same pattern applies: caller has resource-level access, service agent has KMS-level access.

AWS vs. GCP: A Comparison

If you’re coming from AWS: remember that in GCP you don’t grant KMS access to workloads. You grant it once to the service agent.

Fixing the Problem

Here’s the correct way to set things up with Cloud Storage:

1. Find the service agent for your project.
2. Grant it roles/cloudkms.cryptoKeyEncrypterDecrypter on the KMS key.
3. Configure your bucket to use the CMEK by default.

gcloud storage service-agent \
  --project=PROJECT_ID \
  --authorize-cmek=projects/KEY_PROJECT/locations/LOCATION/keyRings/RING/cryptoKeys/KEY

gcloud storage buckets update gs://BUCKET_NAME \
  --default-encryption-key=projects/KEY_PROJECT/locations/LOCATION/keyRings/RING/cryptoKeys/KEY

Once you do this, uploads and downloads work as expected.

PROJECT_NUMBER=$(gcloud projects describe PROJECT_ID --format='value(projectNumber)')
SERVICE_AGENT=service-${PROJECT_NUMBER}@gs-project-accounts.iam.gserviceaccount.com

# Grant the service agent encryption and decryption rights
 gcloud kms keys add-iam-policy-binding projects/KEY_PROJECT/locations/LOCATION/keyRings/RING/cryptoKeys/KEY \
   --member=serviceAccount:${SERVICE_AGENT} \
   --role=roles/cloudkms.cryptoKeyEncrypterDecrypter

Replace PROJECT_ID, KEY_PROJECT, LOCATION, RING, and KEY with your values.

Conclusion

This whole experience started with a simple assumption that GCP would behave like AWS when it came to encryption, but it quickly showed me that the two platforms have very different models.

In AWS, the caller’s role directly interacts with KMS. In GCP, it’s the service agent, not your own service account, that takes care of the encryption and decryption requests to Cloud KMS.

If you are moving from AWS to GCP, keep this in mind: your service account gives you access to the service, while the service agent is what actually talks to KMS. That small but important distinction will save you time and frustration when setting up CMEK across GCP services.