Eks-core-services module run is failing with certificate not found error

Gruntwork Customers
, ,
1

Hi,

We are trying to bring up a EKS cluster in AWS using gruntwork modules. We have followed below procedure.

  1. Run eks-cluster live
  2. Installed kubergrunt and kubectl
  3. Run k8s-applications-namespace live
  4. Run eks-core-services live
https://github.com/gruntwork-io/infrastructure-live-multi-account-acme/tree/master/dev/us-east-1/dev/services

While running eks-core-services it failed with an error saying certificate is not found or there is no permission to access it. The error snippet is given below. Now we are in a state where we are not able to delete eks-core-services!

Can you please confirm whether the order we followed in creating EKS cluster is correct. Also appreciate if you can help us in getting rid of this error and bringing up the cluster.

Error: failed to execute "/usr/local/bin/kubergrunt": time="2020-02-13T11:42:54Z" level=info msg="--kubectl-server-endpoint provided. Checking for --kubectl-certificate-authority and --kubectl-token." name=kubergrunt
time="2020-02-13T11:42:54Z" level=info msg="Received instruction to generate temporary directory as helm home (--helm-home=__TMP__)." name=kubergrunt
time="2020-02-13T11:42:54Z" level=info msg="Generated temporary directory /tmp/696712633/.helm" name=kubergrunt
time="2020-02-13T11:42:54Z" level=info msg="Setting up local helm client to access Tiller server deployed in namespace kube-system." name=kubergrunt
time="2020-02-13T11:42:54Z" level=info msg="Checking if authorized to access specified Tiller server." name=kubergrunt
time="2020-02-13T11:42:54Z" level=info msg="Loading Kubernetes Client" name=kubergrunt
time="2020-02-13T11:42:54Z" level=info msg="Using direct auth methods to setup client." name=kubergrunt
time="2020-02-13T11:42:54Z" level=info msg="Loading Kubernetes Client" name=kubergrunt
time="2020-02-13T11:42:54Z" level=info msg="Using direct auth methods to setup client." name=kubergrunt
time="2020-02-13T11:42:54Z" level=error msg="You do not have permissions to access the client certs for Tiller deployed in namespace kube-system, or they do not exist." name=kubergrunt
ERROR: secrets "tiller-client-2e3a9c6ce8da519bb53fafcea0f28db4-certs" not found


  on main.tf line 56, in data "external" "configured_helm_home":
  56: data "external" "configured_helm_home" {
2

Hi,

Can you share:

  • What version of kubergrunt you are using
  • Which IAM role you are logged in as

By default the eks-core-services module sets up the access permissions to only allow access from the allow-full-access-from-other-accounts IAM role. My guess is that you are not using that role to deploy the infrastructure. It should work once you switch to that IAM role.

Yori

3

Thank you very much for replying. And sorry for the delayed response.
I am using Kubergrunt version v0.5.8
Yes, I have tried this after switching the role to “allow-full-access-from-other-accounts/ec2-user”

sts get-caller-identity 
{
    "Account": "<Acc ID>", 
    "UserId": "<USER ID>:ec2-user", 
    "Arn": "arn:aws:sts::<Acc ID>:assumed-role/allow-full-access-from-other-accounts/ec2-user"
}

Also, after creating tiller instance in separate namespace and if I tried to configure helm client, this is also failing with the same permission error.

kubergrunt helm configure --tiller-namespace applications-tiller --resource-namespace applications --rbac-user applications-tiller-access-all 
INFO[2020-03-03T04:36:53Z] No context name provided. Using default.      name=kubergrunt
INFO[2020-03-03T04:36:53Z] No kube config path provided. Using default (/home/ec2-user/.kube/config)  name=kubergrunt
INFO[2020-03-03T04:36:53Z] Setting up local helm client to access Tiller server deployed in namespace applications-tiller.  name=kubergrunt
INFO[2020-03-03T04:36:53Z] Checking if authorized to access specified Tiller server.  name=kubergrunt
INFO[2020-03-03T04:36:53Z] Loading Kubernetes Client                     name=kubergrunt
INFO[2020-03-03T04:36:53Z] No direct auth methods provided. Using config on disk and context.  name=kubergrunt
INFO[2020-03-03T04:36:53Z] Loading Kubernetes Client                     name=kubergrunt
INFO[2020-03-03T04:36:53Z] No direct auth methods provided. Using config on disk and context.  name=kubergrunt
ERRO[2020-03-03T04:36:53Z] You do not have permissions to access the client certs for Tiller deployed in namespace applications-tiller, or they do not exist.  name=kubergrunt
ERROR: secrets "tiller-client-f47ef214cbd2f8c97b6581cecb7c954a-certs" not found
4

If you are using kubergrunt v0.5.8, you should be able to see which rbac users have access. Run kubectl get secrets -n applications-tiller. There should be a secret with the name tiller-client-RANDOM_HASH-certs. For each one, run kubectl describe secret NAME -n applications-tiller, which will have annotations that indicate which RBAC user/group/service account the client cert is for. One of them should be for the RBAC user allow-full-access-from-other-accounts.

If there isn’t one, you need to grant access to that role:

kubergrunt helm grant --tiller-namespace applications-tiller --rbac-user allow-full-access-from-other-accounts --tls-common-name admin-role --tls-org YOUR_ORG
5

Thank you. I am able to find the right user for allow-full-access-from-other-accounts with the help of above commands.
When I run helm configure it is trying to access some secret which is not shown up in “kubectl get secrets -n applications-tiller” output. And it is failing with the key doesn’t exist error.

I am not sure from where it is fetching this secret.

kubergrunt helm configure --tiller-namespace applications-tiller --resource-namespace applications --rbac-user tiller-client-rbac-user
INFO[2020-03-03T06:46:39Z] No context name provided. Using default.      name=kubergrunt
INFO[2020-03-03T06:46:39Z] No kube config path provided. Using default (/home/ec2-user/.kube/config)  name=kubergrunt
INFO[2020-03-03T06:46:39Z] Setting up local helm client to access Tiller server deployed in namespace applications-tiller.  name=kubergrunt
INFO[2020-03-03T06:46:39Z] Checking if authorized to access specified Tiller server.  name=kubergrunt
INFO[2020-03-03T06:46:39Z] Loading Kubernetes Client                     name=kubergrunt
INFO[2020-03-03T06:46:39Z] No direct auth methods provided. Using config on disk and context.  name=kubergrunt
INFO[2020-03-03T06:46:39Z] Loading Kubernetes Client                     name=kubergrunt
INFO[2020-03-03T06:46:39Z] No direct auth methods provided. Using config on disk and context.  name=kubergrunt
ERRO[2020-03-03T06:46:39Z] You do not have permissions to access the client certs for Tiller deployed in namespace applications-tiller, or they do not exist.  name=kubergrunt
ERROR: secrets "tiller-client-c3e2a2a00ed25bff37d4bc0902c56f42-certs" not found
6

Thank you very much.
The problem is solved and I am able to configure helm. I had granted access for the organization and which in turn generated another secret. After that helm configure worked!

I am going to run ‘eks-core-services’ module and update the result soon.

7

Resolving the ticket as it is worked with above steps.
Thank you!

8

Thanks for closing the loop!