I was using Vault in the following environment.
vault:
https://github.com/hashicorp/vault-helm/releases/tag/v0.5.0
GKE: 1.15.11-gke.5
Backend: GCS
I use
Workload Identity
to access KMS and GCS from the vault server.
This was working well until yesterday (2020/05/11).
I updated GKE version to
1.15.11-gke.12
yesterday, and suddenly vault doesn’t start.
Error Log:
2020-05-12T13:35:04.633Z [WARN] storage migration check error: error="failed to read value for "core/migration": Get https://storage.googleapis.com/xxxx-storage/core/migration: compute: Received 403 `
Unable to generate token; IAM returned 403 Forbidden: Request had insufficient authentication scopes.
This error could be caused by a missing IAM policy binding on the target IAM service account.
You can create the necessary policy binding with:
gcloud iam service-accounts add-iam-policy-binding \
--role=roles/iam.workloadIdentityUser \
--member="serviceAccount:xxxx.svc.id.goog[default/vault]" \
[email protected]
For more information, refer to the Workload Identity documentation:
https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity
I tried reverting the version of GKE, creating a new GKE cluster and installing vault from helm again, but I can’t start it with the same error.
I tried adding full access to GCS to ServiceAccount, but it didn’t work.
I thought it was a Workload Identity issue, so I started POD with the same service account and tried to access GCS from gsutil, and this worked.
I am at a loss for the cause.
Do you have any possible problems?
I am using the same GKE version but its failing with below error after upgrading GKE from to 1.15.9-gke.24,
severity: “ERROR”
textPayload: "2020-05-29T06:40:06.523Z [WARN] storage migration check error: error=“failed to read value for “core/migration”: googleapi: got HTTP response code 403 with body: <?xml version='1.0' encoding='UTF-8'?>AccessDeniedAccess denied.
Primary: /namespaces/project.svc.id.goog with additional claims does not have storage.objects.get access to the Google Cloud Storage object. ”
