Skip to content

how i fixed a production-grade kubernetes secret leak—and how you can avoid it too

how i fixed a production-grade kubernetes secret leak — and how you can avoid it too

when i first started working in devops, i thought securing secrets in kubernetes was simple: just use secret objects and you're done. that’s what i believed — until a misconfigured helm chart exposed our database credentials in a public github repo. the resulting incident took 4 hours to contain, cost us over $12k in remediation, and taught me a hard lesson: secrets in kubernetes are not automatically secure.

what went wrong? the real culprit

here’s what happened:

  • i used helm template to generate manifests locally and committed them to git
  • i didn’t realize that secret values are base64-encoded, not encrypted
  • our ci/cd pipeline automatically deployed these manifests to production
  • a hacker used a simple grep -r "k8s-secret" . on github to find our base64 strings
  • they decoded them with: echo "cgfzc3dvcmqxmjm=" | base64 --decode — and had full db access

i was embarrassed. but more importantly — i learned that encoding ≠ encryption. base64 is for transport, not protection.

the fix: 5 practical steps you can apply today

step 1: never commit secrets to version control

use .gitignore religiously. add these lines:

.env
values.yaml
*secret.yaml
secrets/

step 2: use external secret managers

replace inline secrets with references to tools like:

  • aws secrets manager (with external-secrets operator)
  • vault by hashicorp (integrated via vault-secrets-webhook)
  • azure key vault (via azure csi driver)

example: using external secrets operator to pull from aws:

apiversion: external-secrets.io/v1beta1
kind: externalsecret
metadata:
  name: db-creds
spec:
  secretstoreref:
    name: aws-secrets-store
    kind: secretstore
  target:
    name: db-credentials
  data:
  - secretkey: username
    remoteref:
      key: /prod/db/creds
      property: username
  - secretkey: password
    remoteref:
      key: /prod/db/creds
      property: password

step 3: use kustomize or helm with values override

never hardcode secrets in yaml templates. instead, use overrides:

# values.yaml (safe to commit)
auth:
  username: "placeholder"
  password: "placeholder"
# values-production.yaml (never commit this!)
auth:
  username: "prod-user"
  password: "supersecret123!"

then deploy with:

helm upgrade --install my-app ./my-chart -f values-production.yaml

step 4: enable automatic secrets rotation

tools like vault and external secrets operator can auto-rotate credentials. configure your postgresql user to rotate every 7 days — then redeploy the pod to fetch the new secret.

step 5: scan for secrets with gitguardian or trivy

add a pre-commit hook or ci step to scan for accidentally exposed secrets:

# in your .github/workflows/secret-scan.yaml
- name: scan for secrets
  uses: gitguardian/ggshield@v2
  with:
    args: "scan git -r ."

or use trivy on helm charts:

trivy conf --severity high,critical ./my-chart/

pro tips for full stack and coding teams

  • use short-lived tokens — don’t rely on permanent credentials
  • make review-by-secrets-check mandatory in your pr templates
  • train your frontend and backend devs: “if it’s in git, it’s public — even if it looks encrypted”
  • use sealedsecrets if you must keep secrets in git (e.g., for gitops): they’re encrypted by a key in the cluster

final thoughts: security is a habit, not an event

kubernetes doesn’t make you secure — your team’s practices do. whether you're a full stack developer, a coding student, or a junior devops engineer: make secret handling part of your muscle memory.

next time you edit a yaml file and see a data: field with long base64 strings — stop. ask: “is this supposed to be here? could someone else see this?”

follow these steps, and you won’t be the one writing the post-mortem. you’ll be the one mentoring others on how to avoid the mistake — and that’s the real win.

seo reminder

when writing about kubernetes security, include these key phrases naturally in your文章: devops secrets best practices, kubernetes secret leak fix, secure coding kubernetes, how to protect secrets in helm. these help your guide rank for students and engineers searching for answers — just like you did.

The views expressed in this post are those of its author and do not represent pilput. See our Terms of Service.

Comments

Share your thoughts and join the conversation.

Loading comments...

Join the Discussion

Please log in to share your thoughts and engage with the community.