Tutorial: Encrypting Kubernetes Secrets with Sealed Secrets
Sealed Secrets is a solution to store encrypted Kubernetes secrets in version control.
In this blog post we’ll learn how to install and use it.
Comparison with helm-secrets and sops
A popular alternative to Sealed Secrets is helm-secrets which uses sops as a backend.
The main difference is:
- Sealed Secrets decrypts the secret server-side
- Helm-secrets decrypts the secret client-side
Client-side decryption with helm-secrets can be a security risk since the client (such as a CI/CD system) needs to have access to the encryption key to perform the deployment. Note that this is not a problem if you use GitOps tools such as Argo CD or Flux.
With Sealed Secrets and server-side decryption we can avoid this security risk. The encryption key only exists in the Kubernetes cluster and is never exposed.
Sealed Secrets is not able to use cloud KMS solutions such as AWS KMS. If this is a requirement then go with sops/helm-secrets.
Installation via Helm chart
Sealed Secrets consists of two components:
- Client-side CLI tool to encrypt secrets and create sealed secrets
- Server-side controller used to decrypt sealed secrets and create secrets
To install the controller in our Kubernetes cluster we’ll use the official Helm chart from the sealed-secrets repository.
Add the repository and install it to the
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets helm install sealed-secrets --namespace kube-system --version 1.13.2 sealed-secrets/sealed-secrets
CLI tool installation
Secrets are encrypted client-side using the
kubeseal CLI tool.
For macOS, we can use the Homebrew formula. For Linux, we can download the binary from the GitHub release page.
# macos brew install kubeseal # linux wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.13.1/kubeseal-linux-amd64 -O kubeseal sudo install -m 755 kubeseal /usr/local/bin/kubeseal
kubeseal CLI uses the current
kubectl context to access the cluster. Before continuing make sure that
kubectl is connected to the cluster where Sealed Secrets should be installed.
Creating a sealed secret
kubeseal CLI takes a Kubernetes
Secret manifest as an input, encrypts it and outputs a
In this tutorial we’ll use this secret manifest as an input:
apiVersion: v1 kind: Secret metadata: creationTimestamp: null name: my-secret data: password: YmFy username: Zm9v
Store the manifest in a file named
secret.yaml and encrypt it:
cat secret.yaml | kubeseal \ --controller-namespace kube-system \ --controller-name sealed-secrets \ --format yaml \ > sealed-secret.yaml
The content of the
sealed-secret.yaml file should look like this:
apiVersion: bitnami.com/v1alpha1 kind: SealedSecret metadata: creationTimestamp: null name: my-secret namespace: default spec: encryptedData: password: AgA... username: AgA... template: metadata: creationTimestamp: null name: my-secret namespace: default
We should now have the secret in
secret.yaml and the sealed secret in
Note: It’s not a good practice to store the unencrypted secret in a file. This is only for demonstration purposes in this tutorial.
To deploy the sealed secret we apply the manifest with kubectl:
kubectl apply -f sealed-secret.yaml
The controller in the cluster will notice that a
SealedSecret resource has been created, decrypt it and create a decrypted
Let’s fetch the secret to make sure that the controller has successfully unsealed it:
kubectl get secret my-secret -o yaml
The data should contain our base64 encoded username and password:
... data: password: YmFy username: Zm9v ...
Everything went well. The secret has been successfully unsealed.
Updating a sealed secret
To update a value in a sealed secret, we have to create a new
Secret manifest locally and merge it into an existing
SealedSecret with the
In the example below we update the value of the password key (
my new password.
echo -n "my new password" \ | kubectl create secret generic xxx --dry-run=client --from-file=password=/dev/stdin -o json \ | kubeseal --controller-namespace=kube-system --controller-name=sealed-secrets --format yaml --merge-into sealed-secret.yaml kubectl apply -f sealed-secret.yaml
The local secret is temporary and the name (
xxx in our case) doesn’t matter. The name of the sealed secret will stay the same.
Adding a new value to a sealed secret
The difference between updating a value and adding a new value is the name of the key. If a key named
password already exists, it will update it. If it doesn’t exist, it will add it.
For example to add a new
api_key key (
--from-file=api_key) into our secret we run:
echo -n "my secret api key" \ | kubectl create secret generic xxx --dry-run=client --from-file=api_key=/dev/stdin -o json \ | kubeseal --controller-namespace=kube-system --controller-name=sealed-secrets --format yaml --merge-into sealed-secret.yaml kubectl apply -f sealed-secret.yaml
Deleting a value from a sealed secret
To delete a key from the sealed secret we have to remove it from the YAML file:
# BSD sed (macOS) sed -i '' '/api_key:/d' sealed-secret.yaml # GNU sed sed -i '/api_key:/d' sealed-secret.yaml kubectl apply -f sealed-secret.yaml
After applying the file, the controller will update the
Secret automatically and remove the
Delete the sealed secret
To delete the secret, we use kubectl to delete the resource:
kubectl delete -f sealed-secret.yaml
This will delete the
SealedSecret resource from the cluster as well as the corresponding
Sealed Secrets is a secure way to manage Kubernetes secrets in version control. The encryption key is stored and secrets are decrypted in the cluster. The client doesn’t have access to the encryption key.
The client uses the
kubeseal CLI tool to generate
SealedSecret manifests that hold encrypted data. After applying the file the server-side controller will recognize a new sealed secret resource and decrypt it to create a