463 lines
13 KiB
Markdown
463 lines
13 KiB
Markdown
# External Secrets - Usage Guide
|
|
|
|
This guide explains how to use External Secrets Operator (ESO) to manage secrets across your Kubernetes cluster and sync them from various secret backends (Vault, Kubernetes secrets, etc.) into your applications.
|
|
|
|
## Table of Contents
|
|
|
|
- [Overview](#overview)
|
|
- [Architecture](#architecture)
|
|
- [Getting Started](#getting-started)
|
|
- [Using External Secrets in Your Repository](#using-external-secrets-in-your-repository)
|
|
- [SecretStore Configuration](#secretstore-configuration)
|
|
- [Creating External Secrets](#creating-external-secrets)
|
|
- [Examples](#examples)
|
|
- [Troubleshooting](#troubleshooting)
|
|
|
|
## Overview
|
|
|
|
External Secrets Operator allows you to use external secret management systems (Vault, AWS Secrets Manager, Azure Key Vault, etc.) as a source of truth for secrets in your Kubernetes cluster. It automatically syncs secrets from these backends into native Kubernetes Secrets.
|
|
|
|
**Key Benefits:**
|
|
- Centralized secret management
|
|
- Automatic synchronization
|
|
- No need to store secrets in Git
|
|
- Support for multiple secret backends
|
|
- Templating and transformation capabilities
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ External Secret Operator (ESO) │
|
|
│ (installed in external-secrets namespace) │
|
|
└─────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌───────────────────┼───────────────────┐
|
|
↓ ↓ ↓
|
|
┌─────────┐ ┌─────────┐ ┌──────────┐
|
|
│ Vault │ │ AWS │ │ Kubernetes
|
|
│ Secrets │ │ SecretsM│ │ Secrets
|
|
└─────────┘ └─────────┘ └──────────┘
|
|
↑ ↑ ↑
|
|
└───────────────────┼───────────────────┘
|
|
↓
|
|
┌───────────────────────────────────┐
|
|
│ SecretStore (or ClusterStore) │
|
|
│ (defines backend connection) │
|
|
└───────────────────────────────────┘
|
|
↓
|
|
┌───────────────────────────────────┐
|
|
│ ExternalSecret │
|
|
│ (defines what to sync & where) │
|
|
└───────────────────────────────────┘
|
|
↓
|
|
┌───────────────────────────────────┐
|
|
│ Kubernetes Secret │
|
|
│ (synced and kept in sync) │
|
|
└───────────────────────────────────┘
|
|
↓
|
|
┌───────────────────────────────────┐
|
|
│ Your Application Pods │
|
|
│ (consumes the secrets) │
|
|
└───────────────────────────────────┘
|
|
```
|
|
|
|
## Getting Started
|
|
|
|
### Installation
|
|
|
|
External Secrets Operator is already installed in your cluster via the Helm chart in this repo. Verify the installation:
|
|
|
|
```bash
|
|
kubectl get deployments -n external-secrets
|
|
kubectl get crd | grep external-secrets
|
|
```
|
|
|
|
You should see:
|
|
- Deployment: `external-secrets`
|
|
- Deployment: `external-secrets-webhook`
|
|
- CRDs: `secretstores.external-secrets.io`, `externalsecrets.external-secrets.io`, etc.
|
|
|
|
## Using External Secrets in Your Repository
|
|
|
|
### Directory Structure
|
|
|
|
For each repository that needs to manage secrets with External Secrets, create a `secrets-reponame` folder in your Kubernetes manifests:
|
|
|
|
```
|
|
my-app-repo/
|
|
├── src/
|
|
├── k8s/
|
|
│ ├── base/
|
|
│ ├── overlays/
|
|
│ └── secrets-my-app/ ← Create this folder
|
|
│ ├── secretstore.yaml
|
|
│ ├── externalsecret.yaml
|
|
│ └── values.yaml (optional, for kustomize/helm values)
|
|
└── README.md
|
|
```
|
|
|
|
### Folder Contents
|
|
|
|
The `secrets-reponame` folder should contain:
|
|
|
|
1. **secretstore.yaml** - Defines how to connect to your secret backend (Vault, AWS, etc.)
|
|
2. **externalsecret.yaml** - Defines which secrets to sync from the backend into Kubernetes
|
|
|
|
## SecretStore Configuration
|
|
|
|
A `SecretStore` is a namespaced resource that defines the connection to your secret backend. Create `secretstore.yaml` in your `secrets-reponame` folder:
|
|
|
|
### Example: Vault Backend
|
|
|
|
```yaml
|
|
apiVersion: external-secrets.io/v1beta1
|
|
kind: SecretStore
|
|
metadata:
|
|
name: vault-secretstore
|
|
namespace: default # Change to your app's namespace
|
|
spec:
|
|
provider:
|
|
vault:
|
|
server: "https://vault.example.com:8200"
|
|
path: "secret"
|
|
auth:
|
|
kubernetes:
|
|
mountPath: "kubernetes"
|
|
role: "my-app-role" # Define this in Vault
|
|
```
|
|
|
|
### Example: Kubernetes Secrets Backend
|
|
|
|
If you want to sync secrets from another namespace or use Kubernetes as the backend:
|
|
|
|
```yaml
|
|
apiVersion: external-secrets.io/v1beta1
|
|
kind: SecretStore
|
|
metadata:
|
|
name: k8s-secretstore
|
|
namespace: default
|
|
spec:
|
|
provider:
|
|
kubernetes:
|
|
remoteNamespace: "secrets-management"
|
|
auth:
|
|
serviceAccount:
|
|
name: external-secrets-reader
|
|
```
|
|
|
|
### ClusterSecretStore (Cluster-Wide)
|
|
|
|
If you want the SecretStore to be available cluster-wide, use `ClusterSecretStore` instead:
|
|
|
|
```yaml
|
|
apiVersion: external-secrets.io/v1beta1
|
|
kind: ClusterSecretStore
|
|
metadata:
|
|
name: vault-cluster-store
|
|
spec:
|
|
provider:
|
|
vault:
|
|
server: "https://vault.example.com:8200"
|
|
path: "secret"
|
|
auth:
|
|
kubernetes:
|
|
mountPath: "kubernetes"
|
|
role: "my-app-role"
|
|
```
|
|
|
|
## Creating External Secrets
|
|
|
|
An `ExternalSecret` is a Kubernetes custom resource that defines:
|
|
- Which `SecretStore` to use
|
|
- Which secrets to fetch from the backend
|
|
- How to transform them
|
|
- Where to create the resulting Kubernetes Secret
|
|
|
|
### Basic Example
|
|
|
|
```yaml
|
|
apiVersion: external-secrets.io/v1beta1
|
|
kind: ExternalSecret
|
|
metadata:
|
|
name: app-secrets
|
|
namespace: default
|
|
spec:
|
|
refreshInterval: 1h # Sync every hour
|
|
secretStoreRef:
|
|
name: vault-secretstore
|
|
kind: SecretStore
|
|
target:
|
|
name: app-secrets # Name of the Kubernetes Secret to create
|
|
creationPolicy: Owner
|
|
template:
|
|
engineVersion: v2
|
|
data:
|
|
DATABASE_URL: "{{ .db_url }}"
|
|
API_KEY: "{{ .api_key }}"
|
|
data:
|
|
- secretKey: db_url
|
|
remoteRef:
|
|
key: my-app/database # Key in Vault
|
|
property: url # Property within that key
|
|
- secretKey: api_key
|
|
remoteRef:
|
|
key: my-app/api # Key in Vault
|
|
property: token
|
|
```
|
|
|
|
### Advanced: Multiple Secrets
|
|
|
|
```yaml
|
|
apiVersion: external-secrets.io/v1beta1
|
|
kind: ExternalSecret
|
|
metadata:
|
|
name: multi-secrets
|
|
namespace: default
|
|
spec:
|
|
refreshInterval: 30m
|
|
secretStoreRef:
|
|
name: vault-secretstore
|
|
kind: SecretStore
|
|
target:
|
|
name: multi-secrets
|
|
creationPolicy: Owner
|
|
dataFrom:
|
|
- extract:
|
|
key: my-app/all-secrets # Fetch entire secret object
|
|
```
|
|
|
|
## Examples
|
|
|
|
### Example 1: Django Application
|
|
|
|
**Directory:** `my-django-app/k8s/secrets-django/`
|
|
|
|
**secretstore.yaml:**
|
|
```yaml
|
|
apiVersion: external-secrets.io/v1beta1
|
|
kind: SecretStore
|
|
metadata:
|
|
name: django-vault
|
|
namespace: django
|
|
spec:
|
|
provider:
|
|
vault:
|
|
server: "https://vault.example.com:8200"
|
|
path: "secret"
|
|
auth:
|
|
kubernetes:
|
|
mountPath: "kubernetes"
|
|
role: "django-app"
|
|
```
|
|
|
|
**externalsecret.yaml:**
|
|
```yaml
|
|
apiVersion: external-secrets.io/v1beta1
|
|
kind: ExternalSecret
|
|
metadata:
|
|
name: django-env
|
|
namespace: django
|
|
spec:
|
|
refreshInterval: 1h
|
|
secretStoreRef:
|
|
name: django-vault
|
|
kind: SecretStore
|
|
target:
|
|
name: django-env
|
|
creationPolicy: Owner
|
|
template:
|
|
engineVersion: v2
|
|
data:
|
|
SECRET_KEY: "{{ .secret_key }}"
|
|
DATABASE_URL: "{{ .database_url }}"
|
|
ALLOWED_HOSTS: "{{ .allowed_hosts }}"
|
|
data:
|
|
- secretKey: secret_key
|
|
remoteRef:
|
|
key: django/prod
|
|
property: secret_key
|
|
- secretKey: database_url
|
|
remoteRef:
|
|
key: django/prod
|
|
property: database_url
|
|
- secretKey: allowed_hosts
|
|
remoteRef:
|
|
key: django/prod
|
|
property: allowed_hosts
|
|
```
|
|
|
|
**Deployment usage:**
|
|
```yaml
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: django-app
|
|
namespace: django
|
|
spec:
|
|
template:
|
|
spec:
|
|
containers:
|
|
- name: django
|
|
envFrom:
|
|
- secretRef:
|
|
name: django-env # Reference the synced secret
|
|
```
|
|
|
|
### Example 2: PostgreSQL Database Credentials
|
|
|
|
**secretstore.yaml:**
|
|
```yaml
|
|
apiVersion: external-secrets.io/v1beta1
|
|
kind: SecretStore
|
|
metadata:
|
|
name: postgres-vault
|
|
namespace: databases
|
|
spec:
|
|
provider:
|
|
vault:
|
|
server: "https://vault.example.com:8200"
|
|
path: "secret"
|
|
auth:
|
|
kubernetes:
|
|
mountPath: "kubernetes"
|
|
role: "postgres"
|
|
```
|
|
|
|
**externalsecret.yaml:**
|
|
```yaml
|
|
apiVersion: external-secrets.io/v1beta1
|
|
kind: ExternalSecret
|
|
metadata:
|
|
name: postgres-credentials
|
|
namespace: databases
|
|
spec:
|
|
refreshInterval: 24h
|
|
secretStoreRef:
|
|
name: postgres-vault
|
|
kind: SecretStore
|
|
target:
|
|
name: postgres-credentials
|
|
creationPolicy: Owner
|
|
data:
|
|
- secretKey: username
|
|
remoteRef:
|
|
key: postgres/primary
|
|
property: username
|
|
- secretKey: password
|
|
remoteRef:
|
|
key: postgres/primary
|
|
property: password
|
|
```
|
|
|
|
### Example 3: TLS Certificates
|
|
|
|
```yaml
|
|
apiVersion: external-secrets.io/v1beta1
|
|
kind: ExternalSecret
|
|
metadata:
|
|
name: tls-certs
|
|
namespace: ingress-nginx
|
|
spec:
|
|
refreshInterval: 720h # 30 days
|
|
secretStoreRef:
|
|
name: vault-cluster-store
|
|
kind: ClusterSecretStore
|
|
target:
|
|
name: app-tls
|
|
creationPolicy: Owner
|
|
template:
|
|
engineVersion: v2
|
|
data:
|
|
tls.crt: "{{ .cert }}"
|
|
tls.key: "{{ .key }}"
|
|
data:
|
|
- secretKey: cert
|
|
remoteRef:
|
|
key: tls/app
|
|
property: certificate
|
|
- secretKey: key
|
|
remoteRef:
|
|
key: tls/app
|
|
property: key
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Check ExternalSecret Status
|
|
|
|
```bash
|
|
# Get ExternalSecret status
|
|
kubectl get externalsecret -n <namespace>
|
|
kubectl describe externalsecret <name> -n <namespace>
|
|
|
|
# Check if the synced Kubernetes Secret exists
|
|
kubectl get secret <target-name> -n <namespace>
|
|
```
|
|
|
|
### View Operator Logs
|
|
|
|
```bash
|
|
# Check External Secrets Operator logs
|
|
kubectl logs -n external-secrets deployment/external-secrets
|
|
|
|
# Check webhook logs (if using webhook)
|
|
kubectl logs -n external-secrets deployment/external-secrets-webhook
|
|
```
|
|
|
|
### Common Issues
|
|
|
|
#### 1. ExternalSecret shows "Error" status
|
|
- Check SecretStore connection details (URL, path, role)
|
|
- Verify Vault authentication is configured correctly
|
|
- Check role exists in Vault and has proper permissions
|
|
- Review operator logs for detailed error messages
|
|
|
|
#### 2. Secret is not being created
|
|
- Ensure `refreshInterval` has passed or manually trigger a sync
|
|
- Verify the target name is correct
|
|
- Check RBAC permissions for the operator
|
|
|
|
#### 3. Secret content is empty
|
|
- Verify the remote key path is correct in Vault
|
|
- Ensure the property name exists in the secret
|
|
- Check that the ExternalSecret template is valid
|
|
|
|
#### 4. Authentication failures
|
|
- Verify the ServiceAccount has the `external-secrets` name or matches the configured account
|
|
- Check Vault auth method configuration
|
|
- Ensure Kubernetes auth role is properly configured in Vault
|
|
|
|
### Force Refresh
|
|
|
|
To trigger an immediate refresh of an ExternalSecret:
|
|
|
|
```bash
|
|
kubectl annotate externalsecret <name> \
|
|
-n <namespace> \
|
|
force-sync="$(date +%s)" \
|
|
--overwrite
|
|
```
|
|
|
|
## Best Practices
|
|
|
|
1. **Use ClusterSecretStore** for widely-shared secrets (databases, certs)
|
|
2. **Use namespaced SecretStore** for app-specific secrets
|
|
3. **Set appropriate refreshInterval** - shorter for sensitive data, longer for stable secrets
|
|
4. **Use templating** for complex secret transformations
|
|
5. **Enable RBAC** to restrict which services can access which secrets
|
|
6. **Monitor** ExternalSecret status regularly
|
|
7. **Version control** the SecretStore and ExternalSecret manifests (not the actual secrets)
|
|
8. **Rotate secrets** in your backend independently of the operator
|
|
|
|
## Resources
|
|
|
|
- [Official External Secrets Documentation](https://external-secrets.io/)
|
|
- [Supported Secret Backends](https://external-secrets.io/latest/introduction/overview/#features)
|
|
- [API Reference](https://external-secrets.io/latest/api/secretstore/)
|
|
- [Vault Integration](https://external-secrets.io/latest/provider/vault/)
|
|
|
|
## Support
|
|
|
|
For issues specific to this cluster's External Secrets setup, check the main README or contact the infrastructure team.
|