fix: Configure Harbor to use cert-manager instead of auto-generated certs
- Change Harbor certSource from 'auto' to 'secret' - Reference stable secret name: harbor-ingress - Keep cert-manager.io/cluster-issuer annotation for auto certificate management - Remove harbor-ingress-v2 workaround name - Add cleanup script and documentation This fixes IncorrectIssuer error where Harbor's self-signed CA conflicted with cert-manager's Let's Encrypt certificate management. Resolves: - 502 errors due to TLS configuration conflict - Failed ACME order finalization (orderNotReady) - Certificate stuck in non-Ready state - Duplicate certificate issuance attempts
This commit is contained in:
parent
af484d51c8
commit
798d50ebb0
292
HARBOR-CERT-FIX.md
Normal file
292
HARBOR-CERT-FIX.md
Normal file
@ -0,0 +1,292 @@
|
||||
# Harbor + cert-manager Fix - Complete Solution
|
||||
|
||||
## Root Cause Analysis
|
||||
|
||||
### The Problem
|
||||
Your Harbor deployment had a **certificate issuer conflict**:
|
||||
|
||||
1. **Harbor chart with `certSource: auto`** (default):
|
||||
- Harbor generates its own self-signed CA certificate
|
||||
- Creates the `harbor-ingress` TLS secret with this self-signed cert
|
||||
- Secret managed by Harbor Helm chart
|
||||
|
||||
2. **cert-manager annotation added to ingress**:
|
||||
- `cert-manager.io/cluster-issuer: letsencrypt` tells cert-manager to manage the cert
|
||||
- cert-manager tries to manage the `harbor-ingress` secret
|
||||
- **Conflict**: cert-manager detects the secret was created by a different issuer
|
||||
- Error: `IncorrectIssuer - Secret was previously issued by "Issuer.cert-manager.io/"`
|
||||
|
||||
3. **Nginx annotations on Traefik ingress**:
|
||||
- Old/irrelevant annotations like `nginx.ingress.kubernetes.io/*` present
|
||||
- Causes confusion and is not best practice
|
||||
|
||||
### Result
|
||||
- ACME order fails: `403 urn:ietf:params:acme:error:orderNotReady`
|
||||
- Certificate stuck in `False` (not Ready) state
|
||||
- Harbor accessible but with Harbor's self-signed cert, not Let's Encrypt
|
||||
|
||||
---
|
||||
|
||||
## The Solution
|
||||
|
||||
### Key Changes in `manifests/harbor/values.yaml`
|
||||
|
||||
```yaml
|
||||
expose:
|
||||
type: ingress
|
||||
tls:
|
||||
enabled: true
|
||||
# Changed from "auto" to "secret"
|
||||
# This tells Harbor: "Don't generate your own cert, use an external secret"
|
||||
certSource: secret
|
||||
secret:
|
||||
# Reference the secret that cert-manager will create
|
||||
secretName: "harbor-ingress"
|
||||
|
||||
ingress:
|
||||
className: traefik
|
||||
annotations:
|
||||
# cert-manager will create the Certificate and Secret automatically
|
||||
cert-manager.io/cluster-issuer: letsencrypt
|
||||
# Traefik annotations for HTTPS routing
|
||||
traefik.ingress.kubernetes.io/router.entrypoints: websecure
|
||||
traefik.ingress.kubernetes.io/router.tls: "true"
|
||||
hosts:
|
||||
core: harbor.dvirlabs.com
|
||||
notary: notary.dvirlabs.com
|
||||
```
|
||||
|
||||
### Why This Works
|
||||
|
||||
1. **`certSource: secret`**:
|
||||
- Harbor chart will NOT auto-generate a certificate
|
||||
- Harbor expects the `harbor-ingress` secret to exist (created externally)
|
||||
- Harbor chart won't interfere with cert-manager's secret management
|
||||
|
||||
2. **`cert-manager.io/cluster-issuer: letsencrypt` annotation**:
|
||||
- cert-manager detects this annotation on the Ingress
|
||||
- Automatically creates a `Certificate` resource
|
||||
- Issues certificate via Let's Encrypt using ClusterIssuer
|
||||
- Stores certificate in the `harbor-ingress` secret
|
||||
|
||||
3. **Clean secret ownership**:
|
||||
- cert-manager is the sole owner/manager of the `harbor-ingress` secret
|
||||
- No "IncorrectIssuer" conflict
|
||||
- Certificate renewals handled automatically by cert-manager
|
||||
|
||||
4. **Stable resource names**:
|
||||
- Ingress: `harbor-ingress`
|
||||
- Secret: `harbor-ingress`
|
||||
- Certificate: `harbor-ingress` (auto-created by cert-manager)
|
||||
- No duplicate/workaround names like `harbor-ingress-v2`
|
||||
|
||||
---
|
||||
|
||||
## Why No IncorrectIssuer or Duplicate Orders
|
||||
|
||||
### IncorrectIssuer Prevention
|
||||
- The old `harbor-ingress` secret (created by Harbor's auto CA) is **deleted** before redeployment
|
||||
- cert-manager creates a fresh secret from scratch
|
||||
- No conflict with previous issuers
|
||||
|
||||
### Duplicate Order Prevention
|
||||
- Only ONE source of truth: **cert-manager via ingress annotation**
|
||||
- Harbor chart does NOT create certificates (certSource: secret)
|
||||
- No separate Certificate manifest needed (ingress annotation is cleaner for GitOps)
|
||||
- cert-manager intelligently reuses valid certificates
|
||||
|
||||
### Clean GitOps Management
|
||||
- Single values file controls everything
|
||||
- ArgoCD manages Harbor deployment
|
||||
- cert-manager automatically handles certificate lifecycle
|
||||
- No manual kubectl hacks needed
|
||||
|
||||
---
|
||||
|
||||
## Deployment Steps
|
||||
|
||||
### 1. Review Changes
|
||||
```bash
|
||||
cd ~/OneDrive/Desktop/gitea/dev-tools
|
||||
git diff manifests/harbor/values.yaml
|
||||
```
|
||||
|
||||
### 2. Run the Fix Script
|
||||
```bash
|
||||
bash fix-harbor-cert.sh
|
||||
```
|
||||
|
||||
This script will:
|
||||
- Delete the old Harbor-generated secret
|
||||
- Clean up failed cert-manager resources
|
||||
- Commit and push changes to git
|
||||
- Trigger ArgoCD sync
|
||||
- Monitor certificate issuance
|
||||
|
||||
### 3. Manual Steps (if you prefer)
|
||||
```bash
|
||||
# Delete old resources
|
||||
kubectl delete secret harbor-ingress -n dev-tools
|
||||
kubectl delete certificate harbor-ingress -n dev-tools --ignore-not-found
|
||||
kubectl delete certificaterequest -n dev-tools -l cert-manager.io/certificate-name=harbor-ingress
|
||||
|
||||
# Commit changes
|
||||
git add manifests/harbor/values.yaml
|
||||
git commit -m "fix: Configure Harbor to use cert-manager for TLS"
|
||||
git push
|
||||
|
||||
# Sync ArgoCD
|
||||
kubectl patch app harbor -n argocd --type merge -p '{"operation":{"initiatedBy":{"username":"manual"},"sync":{"revision":"HEAD"}}}'
|
||||
|
||||
# Monitor
|
||||
kubectl get certificate harbor-ingress -n dev-tools -w
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
### Check Certificate Status
|
||||
```bash
|
||||
kubectl get certificate,secret,ingress -n dev-tools | grep harbor-ingress
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
certificate.cert-manager.io/harbor-ingress True harbor-ingress 1m
|
||||
secret/harbor-ingress kubernetes.io/tls 3 1m
|
||||
ingress.networking.k8s.io/harbor-ingress traefik harbor.dvirlabs.com 192.168.10.240 80, 443 1m
|
||||
```
|
||||
|
||||
### Check Certificate Details
|
||||
```bash
|
||||
kubectl describe certificate harbor-ingress -n dev-tools
|
||||
```
|
||||
|
||||
Should show:
|
||||
- `Status: True`
|
||||
- `Message: Certificate is up to date and has not expired`
|
||||
- `Issuer: letsencrypt` (ClusterIssuer)
|
||||
|
||||
### Test Access
|
||||
```bash
|
||||
curl -I https://harbor.dvirlabs.com
|
||||
```
|
||||
|
||||
Should return `HTTP/2 200` (not 502)
|
||||
|
||||
### Verify Certificate in Browser
|
||||
Visit `https://harbor.dvirlabs.com` - should show:
|
||||
- Valid Let's Encrypt certificate
|
||||
- Issued to: harbor.dvirlabs.com
|
||||
- Issued by: Let's Encrypt Authority
|
||||
|
||||
---
|
||||
|
||||
## Alternative: Explicit Certificate Resource
|
||||
|
||||
If you prefer **declarative Certificate management** instead of ingress annotations, you can:
|
||||
|
||||
### Create `manifests/harbor/certificate.yaml`:
|
||||
```yaml
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: harbor-ingress
|
||||
namespace: dev-tools
|
||||
spec:
|
||||
secretName: harbor-ingress
|
||||
issuerRef:
|
||||
name: letsencrypt
|
||||
kind: ClusterIssuer
|
||||
dnsNames:
|
||||
- harbor.dvirlabs.com
|
||||
```
|
||||
|
||||
### Remove ingress annotation:
|
||||
In `manifests/harbor/values.yaml`, remove:
|
||||
```yaml
|
||||
cert-manager.io/cluster-issuer: letsencrypt # Remove this line
|
||||
```
|
||||
|
||||
### Why I recommend ingress annotations instead:
|
||||
1. **Simpler**: No separate Certificate file to manage
|
||||
2. **DRY principle**: Hostname defined once in values.yaml
|
||||
3. **Less resources**: One less manifest file
|
||||
4. **Harbor chart native**: Uses Harbor's standard annotation mechanism
|
||||
|
||||
---
|
||||
|
||||
## Production Checklist
|
||||
|
||||
- [x] Harbor chart uses `certSource: secret` (not "auto")
|
||||
- [x] Secret name is stable: `harbor-ingress`
|
||||
- [x] cert-manager annotation present on ingress
|
||||
- [x] ClusterIssuer `letsencrypt` exists and is ready
|
||||
- [x] Old self-signed secret deleted before redeployment
|
||||
- [x] Traefik-specific annotations only (no nginx annotations)
|
||||
- [x] HTTPS entrypoint configured: `websecure`
|
||||
- [x] externalURL uses HTTPS: `https://harbor.dvirlabs.com`
|
||||
- [x] GitOps workflow preserved (ArgoCD manages Harbor)
|
||||
- [x] No duplicate resource names (no -v2, -copy suffixes)
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Certificate stuck in "Issuing" state
|
||||
```bash
|
||||
# Check certificate details
|
||||
kubectl describe certificate harbor-ingress -n dev-tools
|
||||
|
||||
# Check ACME order
|
||||
kubectl get order -n dev-tools
|
||||
|
||||
# Check ACME challenge
|
||||
kubectl get challenge -n dev-tools
|
||||
```
|
||||
|
||||
### DNS-01 validation fails
|
||||
```bash
|
||||
# Check Cloudflare credentials secret
|
||||
kubectl get secret cloudflare-api-token-secret -n cert-manager
|
||||
|
||||
# Check ClusterIssuer status
|
||||
kubectl describe clusterissuer letsencrypt
|
||||
```
|
||||
|
||||
### Harbor pods healthy but 502 error persists
|
||||
```bash
|
||||
# Check if secret exists and has valid cert
|
||||
kubectl get secret harbor-ingress -n dev-tools -o yaml
|
||||
|
||||
# Restart Traefik to reload ingress config
|
||||
kubectl rollout restart deployment traefik -n kube-system
|
||||
|
||||
# Check Traefik logs
|
||||
kubectl logs -n kube-system -l app.kubernetes.io/name=traefik --tail=50
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
**Before:**
|
||||
- Harbor generates self-signed cert → `harbor-ingress` secret
|
||||
- cert-manager tries to manage same secret → **IncorrectIssuer conflict**
|
||||
- ACME orders fail repeatedly
|
||||
|
||||
**After:**
|
||||
- cert-manager manages certificate lifecycle cleanly
|
||||
- Harbor references the cert-manager-created secret
|
||||
- Single source of truth for TLS management
|
||||
- Clean GitOps workflow with stable resource names
|
||||
|
||||
**Files Changed:**
|
||||
- `manifests/harbor/values.yaml` - TLS configuration fixed
|
||||
|
||||
**No Changes Needed:**
|
||||
- Harbor chart templates (work as designed)
|
||||
- Ingress class (traefik)
|
||||
- ClusterIssuer (letsencrypt)
|
||||
- Secret name (harbor-ingress)
|
||||
66
fix-harbor-cert.sh
Normal file
66
fix-harbor-cert.sh
Normal file
@ -0,0 +1,66 @@
|
||||
#!/bin/bash
|
||||
# Harbor + cert-manager cleanup and fix script
|
||||
# This removes the Harbor-generated certificate and lets cert-manager create a clean one
|
||||
|
||||
set -e
|
||||
|
||||
echo "=== Harbor cert-manager Fix ==="
|
||||
echo ""
|
||||
|
||||
# Step 1: Delete the old Harbor-generated TLS secret
|
||||
echo "1. Deleting existing harbor-ingress secret (Harbor's self-signed cert)..."
|
||||
kubectl delete secret harbor-ingress -n dev-tools --ignore-not-found=true
|
||||
echo " ✓ Secret deleted"
|
||||
echo ""
|
||||
|
||||
# Step 2: Delete any failed cert-manager Certificate resources
|
||||
echo "2. Cleaning up failed cert-manager resources..."
|
||||
kubectl delete certificate harbor-ingress -n dev-tools --ignore-not-found=true
|
||||
kubectl delete certificaterequest -n dev-tools -l cert-manager.io/certificate-name=harbor-ingress --ignore-not-found=true
|
||||
echo " ✓ Old certificates cleaned"
|
||||
echo ""
|
||||
|
||||
# Step 3: Commit and push the fixed values.yaml
|
||||
echo "3. Committing fixed Harbor values to git..."
|
||||
cd "$(dirname "$0")"
|
||||
git add manifests/harbor/values.yaml
|
||||
git commit -m "fix: Configure Harbor to use cert-manager for TLS (secretName: harbor-ingress)"
|
||||
git push
|
||||
echo " ✓ Changes pushed to git"
|
||||
echo ""
|
||||
|
||||
# Step 4: Wait for ArgoCD to sync (or trigger manually)
|
||||
echo "4. Waiting for ArgoCD to sync Harbor application..."
|
||||
sleep 5
|
||||
kubectl patch app harbor -n argocd --type merge -p '{"operation":{"initiatedBy":{"username":"manual"},"sync":{"revision":"HEAD"}}}'
|
||||
echo " ✓ ArgoCD sync triggered"
|
||||
echo ""
|
||||
|
||||
# Step 5: Monitor the certificate issuance
|
||||
echo "5. Monitoring certificate creation..."
|
||||
echo " (This may take 1-2 minutes for DNS-01 validation)"
|
||||
echo ""
|
||||
|
||||
for i in {1..24}; do
|
||||
STATUS=$(kubectl get certificate harbor-ingress -n dev-tools -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo "NotFound")
|
||||
|
||||
if [ "$STATUS" == "True" ]; then
|
||||
echo " ✓ Certificate issued successfully!"
|
||||
break
|
||||
elif [ "$STATUS" == "NotFound" ]; then
|
||||
echo " ⏳ Waiting for certificate to be created... ($i/24)"
|
||||
else
|
||||
echo " ⏳ Certificate status: $STATUS ($i/24)"
|
||||
fi
|
||||
|
||||
sleep 5
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "=== Verification ==="
|
||||
kubectl get certificate harbor-ingress -n dev-tools
|
||||
echo ""
|
||||
kubectl get secret harbor-ingress -n dev-tools
|
||||
echo ""
|
||||
echo "=== Complete! ==="
|
||||
echo "Test Harbor at: https://harbor.dvirlabs.com"
|
||||
@ -1,18 +1,25 @@
|
||||
expose:
|
||||
type: ingress
|
||||
tls:
|
||||
# Enable TLS - cert-manager will manage the certificate
|
||||
enabled: true
|
||||
# Use "secret" to reference an existing/external secret managed by cert-manager
|
||||
# DO NOT use "auto" (Harbor's self-signed CA conflicts with cert-manager)
|
||||
certSource: secret
|
||||
secret:
|
||||
# This secret will be created and managed by cert-manager via the ingress annotation
|
||||
secretName: "harbor-ingress"
|
||||
ingress:
|
||||
className: traefik
|
||||
annotations:
|
||||
# cert-manager annotation - will create the certificate automatically
|
||||
cert-manager.io/cluster-issuer: letsencrypt
|
||||
# Traefik specific annotations for HTTPS routing
|
||||
traefik.ingress.kubernetes.io/router.entrypoints: websecure
|
||||
traefik.ingress.kubernetes.io/router.tls: "true"
|
||||
hosts:
|
||||
core: harbor.dvirlabs.com
|
||||
tls:
|
||||
enabled: true
|
||||
certSource: secret
|
||||
secret:
|
||||
secretName: harbor-ingress-v2
|
||||
notary: notary.dvirlabs.com
|
||||
|
||||
externalURL: https://harbor.dvirlabs.com
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user