infra/charts/external-secrets/USAGE.md

External Secrets - Usage Guide

This guide explains how to use External Secrets Operator (ESO) to manage secrets across your Kubernetes cluster and sync them from various secret backends (Vault, Kubernetes secrets, etc.) into your applications.

Table of Contents

• Overview
• Architecture
• Getting Started
• Using External Secrets in Your Repository
• SecretStore Configuration
• Creating External Secrets
• Examples
• Troubleshooting

Overview

External Secrets Operator allows you to use external secret management systems (Vault, AWS Secrets Manager, Azure Key Vault, etc.) as a source of truth for secrets in your Kubernetes cluster. It automatically syncs secrets from these backends into native Kubernetes Secrets.

Key Benefits:

• Centralized secret management
• Automatic synchronization
• No need to store secrets in Git
• Support for multiple secret backends
• Templating and transformation capabilities

Architecture

┌─────────────────────────────────────────────────────────┐
│         External Secret Operator (ESO)                   │
│         (installed in external-secrets namespace)        │
└─────────────────────────────────────────────────────────┘
                            ↓
        ┌───────────────────┼───────────────────┐
        ↓                   ↓                   ↓
   ┌─────────┐         ┌─────────┐        ┌──────────┐
   │  Vault  │         │   AWS   │        │ Kubernetes
   │ Secrets │         │ SecretsM│        │ Secrets
   └─────────┘         └─────────┘        └──────────┘
        ↑                   ↑                   ↑
        └───────────────────┼───────────────────┘
                            ↓
        ┌───────────────────────────────────┐
        │   SecretStore (or ClusterStore)   │
        │   (defines backend connection)    │
        └───────────────────────────────────┘
                            ↓
        ┌───────────────────────────────────┐
        │       ExternalSecret              │
        │  (defines what to sync & where)   │
        └───────────────────────────────────┘
                            ↓
        ┌───────────────────────────────────┐
        │     Kubernetes Secret             │
        │  (synced and kept in sync)        │
        └───────────────────────────────────┘
                            ↓
        ┌───────────────────────────────────┐
        │    Your Application Pods          │
        │   (consumes the secrets)          │
        └───────────────────────────────────┘

Getting Started

Installation

External Secrets Operator is already installed in your cluster via the Helm chart in this repo. Verify the installation:

kubectl get deployments -n external-secrets
kubectl get crd | grep external-secrets

You should see:

• Deployment: external-secrets
• Deployment: external-secrets-webhook
• CRDs: secretstores.external-secrets.io, externalsecrets.external-secrets.io, etc.

Using External Secrets in Your Repository

Directory Structure

For each repository that needs to manage secrets with External Secrets, create a secrets-reponame folder in your Kubernetes manifests:

my-app-repo/
├── src/
├── k8s/
│   ├── base/
│   ├── overlays/
│   └── secrets-my-app/          ← Create this folder
│       ├── secretstore.yaml
│       ├── externalsecret.yaml
│       └── values.yaml          (optional, for kustomize/helm values)
└── README.md

Folder Contents

The secrets-reponame folder should contain:

1. secretstore.yaml - Defines how to connect to your secret backend (Vault, AWS, etc.)
2. externalsecret.yaml - Defines which secrets to sync from the backend into Kubernetes

SecretStore Configuration

A SecretStore is a namespaced resource that defines the connection to your secret backend. Create secretstore.yaml in your secrets-reponame folder:

Example: Vault Backend

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-secretstore
  namespace: default  # Change to your app's namespace
spec:
  provider:
    vault:
      server: "https://vault.example.com:8200"
      path: "secret"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "my-app-role"  # Define this in Vault

Example: Kubernetes Secrets Backend

If you want to sync secrets from another namespace or use Kubernetes as the backend:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: k8s-secretstore
  namespace: default
spec:
  provider:
    kubernetes:
      remoteNamespace: "secrets-management"
      auth:
        serviceAccount:
          name: external-secrets-reader

ClusterSecretStore (Cluster-Wide)

If you want the SecretStore to be available cluster-wide, use ClusterSecretStore instead:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-cluster-store
spec:
  provider:
    vault:
      server: "https://vault.example.com:8200"
      path: "secret"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "my-app-role"

Creating External Secrets

An ExternalSecret is a Kubernetes custom resource that defines:

• Which SecretStore to use
• Which secrets to fetch from the backend
• How to transform them
• Where to create the resulting Kubernetes Secret

Basic Example

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
  namespace: default
spec:
  refreshInterval: 1h  # Sync every hour
  secretStoreRef:
    name: vault-secretstore
    kind: SecretStore
  target:
    name: app-secrets  # Name of the Kubernetes Secret to create
    creationPolicy: Owner
    template:
      engineVersion: v2
      data:
        DATABASE_URL: "{{ .db_url }}"
        API_KEY: "{{ .api_key }}"
  data:
    - secretKey: db_url
      remoteRef:
        key: my-app/database  # Key in Vault
        property: url         # Property within that key
    - secretKey: api_key
      remoteRef:
        key: my-app/api       # Key in Vault
        property: token

Advanced: Multiple Secrets

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: multi-secrets
  namespace: default
spec:
  refreshInterval: 30m
  secretStoreRef:
    name: vault-secretstore
    kind: SecretStore
  target:
    name: multi-secrets
    creationPolicy: Owner
  dataFrom:
    - extract:
        key: my-app/all-secrets  # Fetch entire secret object

Examples

Example 1: Django Application

Directory: my-django-app/k8s/secrets-django/

secretstore.yaml:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: django-vault
  namespace: django
spec:
  provider:
    vault:
      server: "https://vault.example.com:8200"
      path: "secret"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "django-app"

externalsecret.yaml:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: django-env
  namespace: django
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: django-vault
    kind: SecretStore
  target:
    name: django-env
    creationPolicy: Owner
    template:
      engineVersion: v2
      data:
        SECRET_KEY: "{{ .secret_key }}"
        DATABASE_URL: "{{ .database_url }}"
        ALLOWED_HOSTS: "{{ .allowed_hosts }}"
  data:
    - secretKey: secret_key
      remoteRef:
        key: django/prod
        property: secret_key
    - secretKey: database_url
      remoteRef:
        key: django/prod
        property: database_url
    - secretKey: allowed_hosts
      remoteRef:
        key: django/prod
        property: allowed_hosts

Deployment usage:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: django-app
  namespace: django
spec:
  template:
    spec:
      containers:
      - name: django
        envFrom:
        - secretRef:
            name: django-env  # Reference the synced secret

Example 2: PostgreSQL Database Credentials

secretstore.yaml:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: postgres-vault
  namespace: databases
spec:
  provider:
    vault:
      server: "https://vault.example.com:8200"
      path: "secret"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "postgres"

externalsecret.yaml:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: postgres-credentials
  namespace: databases
spec:
  refreshInterval: 24h
  secretStoreRef:
    name: postgres-vault
    kind: SecretStore
  target:
    name: postgres-credentials
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: postgres/primary
        property: username
    - secretKey: password
      remoteRef:
        key: postgres/primary
        property: password

Example 3: TLS Certificates

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: tls-certs
  namespace: ingress-nginx
spec:
  refreshInterval: 720h  # 30 days
  secretStoreRef:
    name: vault-cluster-store
    kind: ClusterSecretStore
  target:
    name: app-tls
    creationPolicy: Owner
    template:
      engineVersion: v2
      data:
        tls.crt: "{{ .cert }}"
        tls.key: "{{ .key }}"
  data:
    - secretKey: cert
      remoteRef:
        key: tls/app
        property: certificate
    - secretKey: key
      remoteRef:
        key: tls/app
        property: key

Troubleshooting

Check ExternalSecret Status

# Get ExternalSecret status
kubectl get externalsecret -n <namespace>
kubectl describe externalsecret <name> -n <namespace>

# Check if the synced Kubernetes Secret exists
kubectl get secret <target-name> -n <namespace>

View Operator Logs

# Check External Secrets Operator logs
kubectl logs -n external-secrets deployment/external-secrets

# Check webhook logs (if using webhook)
kubectl logs -n external-secrets deployment/external-secrets-webhook

Common Issues

1. ExternalSecret shows "Error" status

• Check SecretStore connection details (URL, path, role)
• Verify Vault authentication is configured correctly
• Check role exists in Vault and has proper permissions
• Review operator logs for detailed error messages

2. Secret is not being created

• Ensure refreshInterval has passed or manually trigger a sync
• Verify the target name is correct
• Check RBAC permissions for the operator

3. Secret content is empty

• Verify the remote key path is correct in Vault
• Ensure the property name exists in the secret
• Check that the ExternalSecret template is valid

4. Authentication failures

• Verify the ServiceAccount has the external-secrets name or matches the configured account
• Check Vault auth method configuration
• Ensure Kubernetes auth role is properly configured in Vault

Force Refresh

To trigger an immediate refresh of an ExternalSecret:

kubectl annotate externalsecret <name> \
  -n <namespace> \
  force-sync="$(date +%s)" \
  --overwrite

Best Practices

1. Use ClusterSecretStore for widely-shared secrets (databases, certs)
2. Use namespaced SecretStore for app-specific secrets
3. Set appropriate refreshInterval - shorter for sensitive data, longer for stable secrets
4. Use templating for complex secret transformations
5. Enable RBAC to restrict which services can access which secrets
6. Monitor ExternalSecret status regularly
7. Version control the SecretStore and ExternalSecret manifests (not the actual secrets)
8. Rotate secrets in your backend independently of the operator

Resources

• Official External Secrets Documentation
• Supported Secret Backends
• API Reference
• Vault Integration

Support

For issues specific to this cluster's External Secrets setup, check the main README or contact the infrastructure team.