For production environments, we highly recommend securing and centralizing the storage and management of your secrets (database access, root token, enterprise key, etc…) in AWS Secrets Manager, then allow your EKS cluster to retrieve those secrets using fine-grained IAM policies.
This section will walk you through the steps to enable your EKS cluster to retrieve secrets from AWS Secrets Manager.
Before You Start #
Make sure you have completed the steps found in the AWS deployment instructions before proceeding.
1. Install The AWS Secrets and Configuration Provider (ASCP) #
To retrieve your secrets through your workloads running on your cluster, you will first need to install:
- A Secrets Store CSI driver
- AWS Secrets Manager and Config Provider
The ASCP works with Amazon Elastic Kubernetes Service (Amazon EKS) 1.17+.
Install the Secrets Store CSI Driver #
Deploy the Secrets Store CSI driver by following the installation steps.
Make sure to enable the Sync as Kubernetes Secret
feature explicitly by setting the helm parameter syncSecret.enabled
to true.
helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system --set syncSecret.enabled=true
Install the AWS Provider #
AWS provider for the Secrets Store CSI Driver allows you to make secrets stored in Secrets Manager appear as files mounted in Kubernetes pods.
kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml
2. Store Pachyderm’s Secrets in Secrets Manager #
In your Secret Manager Console, click on Store a new secret, select the Other type of Secret (for generic secrets), provide the following Key/Value pairs, then choose a secret name.
Secret Key | Description | Value |
---|---|---|
root_token | Root clusterAdmin of your cluster | Any |
postgresql_password | Password to your database | Any |
OAUTH_CLIENT_SECRET | Oauth client secret for Console Required if you set an Enterprise key | Any |
enterprise_license_key | Your enterprise license | Your enterprise License key |
pachd_oauth_client_secret | Oauth client secret for pachd | Any |
enterprise_secret | Needed if you connect to an enterprise server | Any |
Create your secret, then retrieve its arn. It will be needed in the next phase.
3. Grant Your EKS Cluster Access To Your Secrets Manager #
Your cluster has an OpenID Connect issuer URL associated with it. To use IAM roles for service accounts, an IAM OIDC provider must exist for your cluster.
Create an IAM OIDC Provider #
Before granting your EKS pods the proper permissions to access your secrets, you need to create an IAM OIDC provider for your cluster or retrieve the arn of your provider if you already have one created.
Follow the steps in AWS user guide
Example #
eksctl utils associate-iam-oidc-provider --cluster="<cluster-name>"
Create An IAM Policy That Grants Read Access To Your Secret #
- Create a new Policy from your IAM Console
- Select the JSON tab.
- Copy/Paste the following text in the JSON tab
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
],
"Resource": [
<!-- Copy the arn of your secret HERE - see example below>
"arn:aws:secretsmanager:<region>:<account>ãī¸<your secret name>"
]
}
]
}
This policy limits the access to the secrets that your EKS cluster needs to access.
Attach Your Policy To An IAM Role and The Role To Your Service Account #
Create an IAM role and attach the IAM policy that you specified to it.
The role is associated with a Kubernetes service account created in the namespace that you specify (your cluster’s) and annotated with eks.amazonaws.com/role-arn:arn:aws:iam::111122223333:role/my-role-name
.
Example #
eksctl create iamserviceaccount \
--name "<my-service-account>" \
--cluster "<my-cluster>" \
--attach-policy-arn \ "<Copy the arn of your policy HERE>" \
--approve \
--override-existing-serviceaccounts
4. Mount Your Secrets In Your EKS Cluster #
To show secrets in EKS as though they are files on the filesystem, you need to create a SecretProviderClass YAML file that contains information about your secrets as well as information on how to display them in the EKS pod. Use the file provided below and run kubectl apply -f yoursecretclass.yaml
.
The SecretProviderClass must be in the same namespace as the EKS cluster.
metadata:
# Insert your secret name
name: pach-secrets
spec:
provider: aws
parameters:
objects: |
- objectName: "pach-secrets"
objectType: "secretsmanager"
jmesPath:
- path: root_token
objectAlias: root-token
- path: postgresql_password
objectAlias: postgresql-password
- path: OAUTH_CLIENT_SECRET
objectAlias: OAUTH_CLIENT_SECRET
- path: enterprise_license_key
objectAlias: enterprise-license-key
- path: pachd_oauth_client_secret
objectAlias: pachd-oauth-client-secret
- path: enterprise_secret
objectAlias: enterprise-secret
secretObjects:
- data:
- key: root-token
objectName: root-token
secretName: root-token
type: Opaque
- data:
- key: postgresql-password
objectName: postgresql-password
secretName: postgresql-password
type: Opaque
- data:
- key: OAUTH_CLIENT_SECRET
objectName: OAUTH_CLIENT_SECRET
secretName: console-oauth-client-secret
type: Opaque
- data:
- key: enterprise-license-key
objectName: enterprise-license-key
secretName: enterprise-license-key
type: Opaque
- data:
- key: pachd-oauth-client-secret
objectName: pachd-oauth-client-secret
secretName: pachd-oauth-client-secret
type: Opaque
- data:
- key: enterprise-secret
objectName: enterprise-secret
secretName: enterprise-secret
type: Opaque
5. Create A Syncer Pod #
Once your secret class is configured, a pod needs to request the class to trigger the CSI driver and retrieve the secrets in Kubernetes.
Update the file below with your serviceAccountName
and secretProviderClass
before you run a kubectl apply -f syncerpod.yaml
apiVersion: v1
kind: Pod
metadata:
name: secret-syncer
spec:
containers:
- name: secret-syncer
image: k8s.gcr.io/pause
volumeMounts:
- name: secrets-store-inline
mountPath: "/mnt/secrets-store"
readOnly: true
terminationGracePeriodSeconds: 3
serviceAccountName: "<Insert your service account name>"
volumes:
- name: secrets-store-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
# Insert the name of your Secret Provider
secretProviderClass: "pach-secrets"
Run a quick kubectl get all
to check on your new pod.
6. Update Secrets In Pachyderm Values.YAML #
Finally, using the secretName
(s) of your SecretProviderClass above, update Pachyderm’s values.YAML with the list of secrets you will be needing.
Choose the ones that apply to your use case.
deployTarget: LOCAL
global:
postgresql:
postgresqlExistingSecretName: postgresql-password
postgresqlExistingSecretKey: postgresql-password
console:
enabled: true
config:
oauthClientSecretSecretName: console-oauth-client-secret
pachd:
rootTokenSecretName: root-token
enterpriseSecretSecretName: enterprise-secret
oauthClientSecretSecretName: pachd-oauth-client-secret
enterpriseLicenseKeySecretName: enterprise-license-key
activateEnterprise: true
Your Secrets Manager is now configured to provide credential values to your cluster.