Kubernetes secrets encryption and usage using secrets managers vs encryption at rest. Are secrets manager even really secure?

Question

I'm getting into Kubernetes security and I'm looking at various ways to encrypt and use Secrets values in pods but I think I'm not grasping some key concepts. As I understood it, from the cluster security standpoint encrypting secrets should avoid that in case of cluster attack the attacker wouldn't be able to get api keys, access tokens, usernames and passwords, just by simply base64 decode the secrets values.

I'm comparing the use of secrets managers like Vault and Sealed Secrets against enabling Encryption at rest.

I see that with implementing either Vault + Vault Secrets Operator,or Vault + External Secrets, or Sealed Secrets, a normal Secret is generated from encrypted secrets and laying around in the cluster.

From Vault Secrets Operator GitHub

The Operator writes the source Vault secret data directly to the destination Kubernetes Secret, ensuring that any changes made to the source are replicated to the destination over its lifetime.In this way, an application only needs to have access to the destination secret in order to make use of the secret data contained within.

From Seal Secret GitHub I see that their Sealed Secret custom resource will get converted to a normal Kubernetes Secret..

This normal kubernetes secret will appear in the cluster after a few seconds you can use it as you would use any secret that you would have created directly (e.g. reference it from a Pod).

Encryption at rest on the other hand, will actually encrypt the secrets upon creation, dough you need to also configure RBAC rules, or use envelope encryption with a third-party KMS provider like Azure Key Vaults for example.

The three methods are almost equally complicated to implement as for example Vault needs a lot of manual configuration for unsealing and create encrypted secrets for example, but only encryption at rest will actually secure sensitive data against cluster attacks as Secrets are actually encrypted and decrypted when used from pods.

Given that above considerations, what are Vault and Sealed Secret good for, and what's even the point of going through all that trouble for setting them up if then Secrets end up laying around unencrypted?

Update - I love this security thing, I really do -

As suggested by David, passing sensitive data to the containerised Node.js app environment through Pod's container environment variables just makes the easy to get, so they're better off in a secure store like Azure Key Vault and retrieved directly in app via the SDK. Fair enough..just update the app to us that. But just there it all starts again.

You now have to secure the KEYVAULT_URI, AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET.

This is the exact situation I thought of when I first started programming and started using .env file not to hard code api keys and other sensitive data and avoid pushing it to the repo. But then you have to pass those values to the containerised app. No worries, there you have a Secret to inject them.. but no..that way with a simple exec command on the pod an attacker could get them all, So you gotta stored them in a secure remote encryted and re-encrypted NASA level service on Mars, but hey.. You need access to them so there you have a bounce of keys, secrets and ids to pass to an SDK in your app to do that.. This way it takes just an extra step to get all securely stored sensitive data. Instead of cracking your app and get them, they need to crack your app and get the keys to go and get them.. oh I almost forgot. Thanks for your Azure credentials too..very kind of yours.

This whole security thing is just a sort of a treasure hunt where a clue takes you to the next one and away to the treasure.. Is there an end to all this security, encryption, key rotation thing? I mean one way or another sensitive data seem to get exposed somewhere.. I might just leave the keys in the car..

Seriously.. how you manage all this? Sorry for the rant.. hope I made you at least smile

Answer 1

Getting the Secret into the cluster at all can be a challenge. You can use Kubernetes RBAC to limit the Secret's visibility, which may help some of the security concerns.

I'd suggest there are three basic levels here:

1. The secret values don't exist in Kubernetes at all, but the application directly integrates with Vault or a similar service. (Hardest to set up and run, especially in non-production.)

2. Kubernetes Secrets exist but they're populated by an operator or integration.

3. Kubernetes Secrets are created at deploy time via kubectl apply or Helm.

If the Secret exists at all, as you note, it's fairly straightforward to get its value. kubectl get secret will have it in all but plain text, kubectl exec can find it in the running Pod, if you can kubectl run a new Pod or create any of the workload-type resources then you can mount the Secret into a new Pod and print out its value. That's an argument for having the application tie directly into the secret store, but that's a more complex setup.

Let's say you're not hard-wired into Vault and need to provide a database password as an environment variable. Where does it actually live; when you deploy the application, how does it get set? One option is to put the password in your CI system's credentials store, but this is a very "leaky" option – the CI script and every step along the chain can see the secret, and you need an administrator of the CI system to create or modify the value.

This is where the various secret-manager tools come in. Sealed Secrets lets you commit an encrypted secret file to source control, so you don't need to coördinate with the CI system to create or update a credential. External Secrets creates a Secret from a system like Vault or AWS Secrets Manager. Vault has a fairly rich access-control system, so this makes it possible for a user to create a secret, and list existing secrets, but not directly retrieve the secret value (at least, not outside of Kubernetes).

So, if you install the secret through the CI system:

credential (only CI admins can update)
    v
CI system --> helm install --> Secret object --> container --> application
    ^              ^                 ^               ^              ^
    credential is visible everywhere

If you let an operator like Sealed Secrets or External Secrets create the Secret object:

                secret store --> operator
                                     v
CI system --> helm install --> Secret object --> container --> application
                                     ^               ^              ^
                                  Kubernetes operations can get credential

And if you change the application code to directly wire into the secret store (Vault, AWS Secrets Manager, ...):

                                                              secret store
                                                                    ^
CI system --> helm install --> Secret object --> container --> application
                                                                    ^
                           credential only visible inside application code
                                  requires wiring specific to secret store