---
title: "how i fixed a production-grade kubernetes secret leak—and how you can avoid it too"
author: "pilput"
canonical: "https://pilput.net/pilput/how-i-fixed-a-production-grade-kubernetes-secret-leakand-how-you-can-avoid-it-too"
published: "2025-09-12T04:21:10.818038Z"
updated: "2025-09-12T04:21:26.203421Z"
description: "how i fixed a production-grade kubernetes secret leak — and how you can avoid it too when i first started working in devops , i thought securing secrets in kub..."
---
# how i fixed a production-grade kubernetes secret leak—and how you can avoid it too

## how i fixed a production-grade kubernetes secret leak — and how you can avoid it too

when i first started working in **devops**, i thought securing secrets in kubernetes was simple: just use `secret` objects and you're done. that’s what i believed — until a misconfigured helm chart exposed our database credentials in a public github repo. the resulting incident took 4 hours to contain, cost us over $12k in remediation, and taught me a hard lesson: **secrets in kubernetes are not automatically secure**.

### what went wrong? the real culprit

here’s what happened:

- i used `helm template` to generate manifests locally and committed them to git

- i didn’t realize that `secret` values are **base64-encoded**, not encrypted

- our ci/cd pipeline automatically deployed these manifests to production

- a hacker used a simple `grep -r "k8s-secret" .` on github to find our base64 strings

- they decoded them with: `echo "cgfzc3dvcmqxmjm=" | base64 --decode` — and had full db access

i was embarrassed. but more importantly — i learned that **encoding ≠ encryption**. base64 is for transport, not protection.

### the fix: 5 practical steps you can apply today

#### step 1: never commit secrets to version control

use `.gitignore` religiously. add these lines:

```
.env
values.yaml
*secret.yaml
secrets/
```

#### step 2: use external secret managers

replace inline secrets with references to tools like:

- **aws secrets manager** (with `external-secrets` operator)

- **vault by hashicorp** (integrated via `vault-secrets-webhook`)

- **azure key vault** (via azure csi driver)

example: using external secrets operator to pull from aws:

```
apiversion: external-secrets.io/v1beta1
kind: externalsecret
metadata:
  name: db-creds
spec:
  secretstoreref:
    name: aws-secrets-store
    kind: secretstore
  target:
    name: db-credentials
  data:
  - secretkey: username
    remoteref:
      key: /prod/db/creds
      property: username
  - secretkey: password
    remoteref:
      key: /prod/db/creds
      property: password
```

#### step 3: use kustomize or helm with values override

never hardcode secrets in yaml templates. instead, use overrides:

```
# values.yaml (safe to commit)
auth:
  username: "placeholder"
  password: "placeholder"
```

```
# values-production.yaml (never commit this!)
auth:
  username: "prod-user"
  password: "supersecret123!"
```

then deploy with:

```
helm upgrade --install my-app ./my-chart -f values-production.yaml
```

#### step 4: enable automatic secrets rotation

tools like **vault** and **external secrets operator** can auto-rotate credentials. configure your postgresql user to rotate every 7 days — then redeploy the pod to fetch the new secret.

#### step 5: scan for secrets with gitguardian or trivy

add a pre-commit hook or ci step to scan for accidentally exposed secrets:

```
# in your .github/workflows/secret-scan.yaml
- name: scan for secrets
  uses: gitguardian/ggshield@v2
  with:
    args: "scan git -r ."
```

or use trivy on helm charts:

```
trivy conf --severity high,critical ./my-chart/
```

### pro tips for full stack and coding teams

- **use short-lived tokens** — don’t rely on permanent credentials

- make `review-by-secrets-check` mandatory in your pr templates

- train your **frontend** and **backend** devs: “if it’s in git, it’s public — even if it looks encrypted”

- use **sealedsecrets** if you must keep secrets in git (e.g., for gitops): they’re encrypted by a key in the cluster

### final thoughts: security is a habit, not an event

kubernetes doesn’t make you secure — **your team’s practices** do. whether you're a **full stack** developer, a **coding** student, or a junior **devops** engineer: make secret handling part of your muscle memory.

next time you edit a yaml file and see a `data:` field with long base64 strings — stop. ask: “is this supposed to be here? could someone else see this?”

follow these steps, and you won’t be the one writing the post-mortem. you’ll be the one mentoring others on how to avoid the mistake — and that’s the real win.

### seo reminder

when writing about kubernetes security, include these key phrases naturally in your文章: **devops secrets best practices**, **kubernetes secret leak fix**, **secure coding kubernetes**, **how to protect secrets in helm**. these help your guide rank for students and engineers searching for answers — just like you did.