Kubernetes auth method : Permission Denied error – HashiCorp Help Center

This article outlines the configuration needed to help fix the permission denied error in k8s auth

Introduction

The kubernetes auth method can be used to authenticate with Vault using a Kubernetes Service Account Token. This method of authentication makes it easy to introduce a Vault token into a Kubernetes Pod.

Problem

When a pod attempts to authenticate to vault via the configured kubernetes auth method; it gets Permission Denied error

Cause

When a pod tried to authenticate to vault and you see an error like this in the audit logs of the vault pod:

{"time":"2020-05-23T00:16:35.5749466Z","type":"response","auth":{"token_type":"default"},"request":{"id":"6f594d4c-40b2-dbfc-d4ba-ef53d1ba03f9","operation":"update","mount_type":"kubernetes","namespace":{"id":"root"},"path":"auth/k8s-auth/login","data":{"jwt":"---BLAH---","role":"test"},"remote_address":"10.121.22.67"},"response":{"mount_type":"kubernetes"},"error":"permission denied"}

This error message is usually caused by one of five reasons:

The JWT used for authenticating to vault does not have the right role associated to it specified during the login

The JWT does not belong to the right service account

The role in k8s auth is not configured properly

The service account does not have the right cluster role binding associated with it

The Vault policy associated with the role does not have the permissions to the path being requested on the Vault side

Overview of possible solutions (if applicable)

Solutions:

Decode your JWT being used for login using jwt.io and examine the payload data to make sure the service account and namespace look right

Below is an example configuration of a working k8s auth. In this below example:

vault-auth is the service account with which k8s auth is configured.


    test-cloud

is the service account used to login to vault using the k8s auth method

test is the namespace of the service account test-cloud


    role-tokenreview-binding

is the name of the cluster role binding the service accounts (vault-auth and test-cloud) need to be associated with

# Create k8s namespace test

$ kubectl create namespace test
namespace/test created

# Create a service account, secret and ClusterRoleBinding with the necessary permissions to allow the service account test-cloud to perform token reviews with k8s

cat <<EOF | kubectl create -f -
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: test-cloud
  namespace: test
---
apiVersion: v1
kind: Secret
metadata:
 name: test-cloud
 namespace: test
 annotations:
   kubernetes.io/service-account.name: test-cloud
type: kubernetes.io/service-account-token
EOF
serviceaccount/test-cloud created
secret/test-cloud created

# Create a service account, secret and ClusterRoleBinding with the necessary permissions to allow vault to perform token reviews with k8s (Make note that clusterRoleBinding has been created associating serviceaccount test-cloud as well)

cat <<EOF | kubectl create -f -
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-auth
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: default
  - kind: ServiceAccount
    name: test-cloud
    namespace: test
EOF
serviceaccount/vault-auth created
secret/vault-auth created
clusterrolebinding.rbac.authorization.k8s.io/role-tokenreview-binding created

# Enable k8s auth method in vault

$ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

# Get the JSON web token (JWT) for vault-auth service account in default namespace to be used by vault k8s config

$ TOKEN_REVIEW_JWT=$(kubectl get secret vault-auth -o go-template='{{ .data.token }}' | base64 --decode)

# Retrieve the k8s CA certificate

$ KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 --decode)
# Retrieve the k8s host URL
$ KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')
# Configure the k8s auth method to use the above sa token, location of the k8s host and its certificate from above
$ vault write auth/kubernetes/config   token_reviewer_jwt="$TOKEN_REVIEW_JWT"   kubernetes_host="$KUBE_HOST"   kubernetes_ca_cert="$KUBE_CA_CERT" disable_local_ca_jwt="true"
Success! Data written to: auth/kubernetes/config
# Read the k8s config
$ vault read auth/kubernetes/config
Key                       Value
---                       -----
disable_iss_validation    false
disable_local_ca_jwt      true
issuer                    n/a
kubernetes_ca_cert        -----BEGIN CERTIFICATE-----
MIIC5zCCAc+gAwIBAgIBATANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwptaW5p
a3ViZUNBMB4XDTIxMDYyMDE0MzA1OVoXDTMxMDYxOTE0MzA1OVowFTETMBEGA1UE
AxMKbWluaWt1YmVDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALZV
jjkUtgyFX/rodABD7SqRPuygBpSik4u2wHLm5Wr101+1aZ+nO7L63ykqAbOL89Nm
-----END CERTIFICATE-----
kubernetes_host           https://172.17.0.61:8443
pem_keys                  []
# Write a policy to associate with the devweb-app role used for login by test-cloud service account
vault policy write devwebapp - <<EOF
path "secret/data/devwebapp/config" {
  capabilities = ["read”]
}
EOF

Success! Uploaded policy: devwebapp
# Associate the role to the serviceaccount and the policy:
vault write auth/kubernetes/role/devweb-app \
  bound_service_account_names=test-cloud \
  bound_service_account_namespaces=test \
  policies=devwebapp \
  ttl=24h
Success! Data written to: auth/kubernetes/role/devweb-app
# Retrieve the JWT for test-cloud sa to login
TOKEN_REVIEW_SJWT=$(kubectl get secret  test-cloud -n test -o go-template='{{ .data.token }}' | base64 --decode)
# Perform a k8s login using test-cloud serviceaccount's JWT and the role
$  curl \
>     --request POST \
>     --data '{"jwt": "'$TOKEN_REVIEW_SJWT'", "role": "devweb-app"}' \
>     http://127.0.0.1:8200/v1/auth/kubernetes/login
{"request_id":"80cac595-c206-348f-2da5-29fe029eddd7","lease_id":"","renewable":false,"lease_duration":0,"data":null,"wrap_info":null,"warnings":null,"auth":{"client_token":"s.mBUJCVNNFKtUVuWVVgfPi9Sj","accessor":"4NoXtX7F9bcu19RZ7TAbtcjT","policies":["default","devwebapp"],"token_policies":["default","devwebapp"],"metadata":{"role":"devweb-app","service_account_name":"test-cloud","service_account_namespace":"test","service_account_secret_name":"test-cloud","service_account_uid":"6d475e37-0288-4070-ad8a-e44abc87b496"},"lease_duration":86400,"renewable":true,"entity_id":"841af42c-87ff-8441-b2d5-0c02ea062384","token_type":"service","orphan":true}}

The login is successful.
If the login is successful, that indicates everything is set up right.
Additional Information
Kubernetes auth with external vault
Kubernetes Auth Method