dev-tools/HARBOR-CERT-FIX.md

Harbor + cert-manager Fix - Complete Solution

Root Cause Analysis

The Problem

Your Harbor deployment had a certificate issuer conflict:

1. Harbor chart with certSource: auto (default):

   • Harbor generates its own self-signed CA certificate
   • Creates the harbor-ingress TLS secret with this self-signed cert
   • Secret managed by Harbor Helm chart

2. cert-manager annotation added to ingress:

   • cert-manager.io/cluster-issuer: letsencrypt tells cert-manager to manage the cert
   • cert-manager tries to manage the harbor-ingress secret
   • Conflict: cert-manager detects the secret was created by a different issuer
   • Error: IncorrectIssuer - Secret was previously issued by "Issuer.cert-manager.io/"

3. Nginx annotations on Traefik ingress:

   • Old/irrelevant annotations like nginx.ingress.kubernetes.io/* present
   • Causes confusion and is not best practice

Result

• ACME order fails: 403 urn:ietf:params:acme:error:orderNotReady
• Certificate stuck in False (not Ready) state
• Harbor accessible but with Harbor's self-signed cert, not Let's Encrypt

---

The Solution

Key Changes in manifests/harbor/values.yaml

expose:
  type: ingress
  tls:
    enabled: true
    # Changed from "auto" to "secret"
    # This tells Harbor: "Don't generate your own cert, use an external secret"
    certSource: secret
    secret:
      # Reference the secret that cert-manager will create
      secretName: "harbor-ingress"
  
  ingress:
    className: traefik
    annotations:
      # cert-manager will create the Certificate and Secret automatically
      cert-manager.io/cluster-issuer: letsencrypt
      # Traefik annotations for HTTPS routing
      traefik.ingress.kubernetes.io/router.entrypoints: websecure
      traefik.ingress.kubernetes.io/router.tls: "true"
    hosts:
      core: harbor.dvirlabs.com
      notary: notary.dvirlabs.com

Why This Works

1. certSource: secret:

   • Harbor chart will NOT auto-generate a certificate
   • Harbor expects the harbor-ingress secret to exist (created externally)
   • Harbor chart won't interfere with cert-manager's secret management

2. cert-manager.io/cluster-issuer: letsencrypt annotation:

   • cert-manager detects this annotation on the Ingress
   • Automatically creates a Certificate resource
   • Issues certificate via Let's Encrypt using ClusterIssuer
   • Stores certificate in the harbor-ingress secret

3. Clean secret ownership:

   • cert-manager is the sole owner/manager of the harbor-ingress secret
   • No "IncorrectIssuer" conflict
   • Certificate renewals handled automatically by cert-manager

4. Stable resource names:

   • Ingress: harbor-ingress
   • Secret: harbor-ingress
   • Certificate: harbor-ingress (auto-created by cert-manager)
   • No duplicate/workaround names like harbor-ingress-v2

---

Why No IncorrectIssuer or Duplicate Orders

IncorrectIssuer Prevention

• The old harbor-ingress secret (created by Harbor's auto CA) is deleted before redeployment
• cert-manager creates a fresh secret from scratch
• No conflict with previous issuers

Duplicate Order Prevention

• Only ONE source of truth: cert-manager via ingress annotation
• Harbor chart does NOT create certificates (certSource: secret)
• No separate Certificate manifest needed (ingress annotation is cleaner for GitOps)
• cert-manager intelligently reuses valid certificates

Clean GitOps Management

• Single values file controls everything
• ArgoCD manages Harbor deployment
• cert-manager automatically handles certificate lifecycle
• No manual kubectl hacks needed

---

Deployment Steps

1. Review Changes

cd ~/OneDrive/Desktop/gitea/dev-tools
git diff manifests/harbor/values.yaml

2. Run the Fix Script

bash fix-harbor-cert.sh

This script will:

• Delete the old Harbor-generated secret
• Clean up failed cert-manager resources
• Commit and push changes to git
• Trigger ArgoCD sync
• Monitor certificate issuance

3. Manual Steps (if you prefer)

# Delete old resources
kubectl delete secret harbor-ingress -n dev-tools
kubectl delete certificate harbor-ingress -n dev-tools --ignore-not-found
kubectl delete certificaterequest -n dev-tools -l cert-manager.io/certificate-name=harbor-ingress

# Commit changes
git add manifests/harbor/values.yaml
git commit -m "fix: Configure Harbor to use cert-manager for TLS"
git push

# Sync ArgoCD
kubectl patch app harbor -n argocd --type merge -p '{"operation":{"initiatedBy":{"username":"manual"},"sync":{"revision":"HEAD"}}}'

# Monitor
kubectl get certificate harbor-ingress -n dev-tools -w

---

Verification

Check Certificate Status

kubectl get certificate,secret,ingress -n dev-tools | grep harbor-ingress

Expected output:

certificate.cert-manager.io/harbor-ingress   True    harbor-ingress   1m
secret/harbor-ingress                        kubernetes.io/tls   3      1m
ingress.networking.k8s.io/harbor-ingress     traefik   harbor.dvirlabs.com   192.168.10.240   80, 443   1m

Check Certificate Details

kubectl describe certificate harbor-ingress -n dev-tools

Should show:

• Status: True
• Message: Certificate is up to date and has not expired
• Issuer: letsencrypt (ClusterIssuer)

Test Access

curl -I https://harbor.dvirlabs.com

Should return HTTP/2 200 (not 502)

Verify Certificate in Browser

Visit https://harbor.dvirlabs.com - should show:

• Valid Let's Encrypt certificate
• Issued to: harbor.dvirlabs.com
• Issued by: Let's Encrypt Authority

---

Alternative: Explicit Certificate Resource

If you prefer declarative Certificate management instead of ingress annotations, you can:

Create manifests/harbor/certificate.yaml:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: harbor-ingress
  namespace: dev-tools
spec:
  secretName: harbor-ingress
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer
  dnsNames:
    - harbor.dvirlabs.com

Remove ingress annotation:

In manifests/harbor/values.yaml, remove:

cert-manager.io/cluster-issuer: letsencrypt  # Remove this line

Why I recommend ingress annotations instead:

1. Simpler: No separate Certificate file to manage
2. DRY principle: Hostname defined once in values.yaml
3. Less resources: One less manifest file
4. Harbor chart native: Uses Harbor's standard annotation mechanism

---

Production Checklist

• Harbor chart uses certSource: secret (not "auto")
• Secret name is stable: harbor-ingress
• cert-manager annotation present on ingress
• ClusterIssuer letsencrypt exists and is ready
• Old self-signed secret deleted before redeployment
• Traefik-specific annotations only (no nginx annotations)
• HTTPS entrypoint configured: websecure
• externalURL uses HTTPS: https://harbor.dvirlabs.com
• GitOps workflow preserved (ArgoCD manages Harbor)
• No duplicate resource names (no -v2, -copy suffixes)

---

Troubleshooting

Certificate stuck in "Issuing" state

# Check certificate details
kubectl describe certificate harbor-ingress -n dev-tools

# Check ACME order
kubectl get order -n dev-tools

# Check ACME challenge
kubectl get challenge -n dev-tools

DNS-01 validation fails

# Check Cloudflare credentials secret
kubectl get secret cloudflare-api-token-secret -n cert-manager

# Check ClusterIssuer status
kubectl describe clusterissuer letsencrypt

Harbor pods healthy but 502 error persists

# Check if secret exists and has valid cert
kubectl get secret harbor-ingress -n dev-tools -o yaml

# Restart Traefik to reload ingress config
kubectl rollout restart deployment traefik -n kube-system

# Check Traefik logs
kubectl logs -n kube-system -l app.kubernetes.io/name=traefik --tail=50

---

Summary

Before:

• Harbor generates self-signed cert → harbor-ingress secret
• cert-manager tries to manage same secret → IncorrectIssuer conflict
• ACME orders fail repeatedly

After:

• cert-manager manages certificate lifecycle cleanly
• Harbor references the cert-manager-created secret
• Single source of truth for TLS management
• Clean GitOps workflow with stable resource names

Files Changed:

• manifests/harbor/values.yaml - TLS configuration fixed

No Changes Needed:

• Harbor chart templates (work as designed)
• Ingress class (traefik)
• ClusterIssuer (letsencrypt)
• Secret name (harbor-ingress)