dev-tools/HARBOR-CERT-FIX.md

# Harbor + cert-manager Fix - Complete Solution

## Root Cause Analysis

### The Problem
Your Harbor deployment had a **certificate issuer conflict**:

1. **Harbor chart with `certSource: auto`** (default):
   - Harbor generates its own self-signed CA certificate
   - Creates the `harbor-ingress` TLS secret with this self-signed cert
   - Secret managed by Harbor Helm chart

2. **cert-manager annotation added to ingress**:
   - `cert-manager.io/cluster-issuer: letsencrypt` tells cert-manager to manage the cert
   - cert-manager tries to manage the `harbor-ingress` secret
   - **Conflict**: cert-manager detects the secret was created by a different issuer
   - Error: `IncorrectIssuer - Secret was previously issued by "Issuer.cert-manager.io/"`

3. **Nginx annotations on Traefik ingress**:
   - Old/irrelevant annotations like `nginx.ingress.kubernetes.io/*` present
   - Causes confusion and is not best practice

### Result
- ACME order fails: `403 urn:ietf:params:acme:error:orderNotReady`
- Certificate stuck in `False` (not Ready) state
- Harbor accessible but with Harbor's self-signed cert, not Let's Encrypt

---

## The Solution

### Key Changes in `manifests/harbor/values.yaml`

```yaml
expose:
  type: ingress
  tls:
    enabled: true
    # Changed from "auto" to "secret"
    # This tells Harbor: "Don't generate your own cert, use an external secret"
    certSource: secret
    secret:
      # Reference the secret that cert-manager will create
      secretName: "harbor-ingress"
  
  ingress:
    className: traefik
    annotations:
      # cert-manager will create the Certificate and Secret automatically
      cert-manager.io/cluster-issuer: letsencrypt
      # Traefik annotations for HTTPS routing
      traefik.ingress.kubernetes.io/router.entrypoints: websecure
      traefik.ingress.kubernetes.io/router.tls: "true"
    hosts:
      core: harbor.dvirlabs.com
      notary: notary.dvirlabs.com
```

### Why This Works

1. **`certSource: secret`**:
   - Harbor chart will NOT auto-generate a certificate
   - Harbor expects the `harbor-ingress` secret to exist (created externally)
   - Harbor chart won't interfere with cert-manager's secret management

2. **`cert-manager.io/cluster-issuer: letsencrypt` annotation**:
   - cert-manager detects this annotation on the Ingress
   - Automatically creates a `Certificate` resource
   - Issues certificate via Let's Encrypt using ClusterIssuer
   - Stores certificate in the `harbor-ingress` secret

3. **Clean secret ownership**:
   - cert-manager is the sole owner/manager of the `harbor-ingress` secret
   - No "IncorrectIssuer" conflict
   - Certificate renewals handled automatically by cert-manager

4. **Stable resource names**:
   - Ingress: `harbor-ingress` 
   - Secret: `harbor-ingress`
   - Certificate: `harbor-ingress` (auto-created by cert-manager)
   - No duplicate/workaround names like `harbor-ingress-v2`

---

## Why No IncorrectIssuer or Duplicate Orders

### IncorrectIssuer Prevention
- The old `harbor-ingress` secret (created by Harbor's auto CA) is **deleted** before redeployment
- cert-manager creates a fresh secret from scratch
- No conflict with previous issuers

### Duplicate Order Prevention
- Only ONE source of truth: **cert-manager via ingress annotation**
- Harbor chart does NOT create certificates (certSource: secret)
- No separate Certificate manifest needed (ingress annotation is cleaner for GitOps)
- cert-manager intelligently reuses valid certificates

### Clean GitOps Management
- Single values file controls everything
- ArgoCD manages Harbor deployment
- cert-manager automatically handles certificate lifecycle
- No manual kubectl hacks needed

---

## Deployment Steps

### 1. Review Changes
```bash
cd ~/OneDrive/Desktop/gitea/dev-tools
git diff manifests/harbor/values.yaml
```

### 2. Run the Fix Script
```bash
bash fix-harbor-cert.sh
```

This script will:
- Delete the old Harbor-generated secret
- Clean up failed cert-manager resources
- Commit and push changes to git
- Trigger ArgoCD sync
- Monitor certificate issuance

### 3. Manual Steps (if you prefer)
```bash
# Delete old resources
kubectl delete secret harbor-ingress -n dev-tools
kubectl delete certificate harbor-ingress -n dev-tools --ignore-not-found
kubectl delete certificaterequest -n dev-tools -l cert-manager.io/certificate-name=harbor-ingress

# Commit changes
git add manifests/harbor/values.yaml
git commit -m "fix: Configure Harbor to use cert-manager for TLS"
git push

# Sync ArgoCD
kubectl patch app harbor -n argocd --type merge -p '{"operation":{"initiatedBy":{"username":"manual"},"sync":{"revision":"HEAD"}}}'

# Monitor
kubectl get certificate harbor-ingress -n dev-tools -w
```

---

## Verification

### Check Certificate Status
```bash
kubectl get certificate,secret,ingress -n dev-tools | grep harbor-ingress
```

Expected output:
```
certificate.cert-manager.io/harbor-ingress   True    harbor-ingress   1m
secret/harbor-ingress                        kubernetes.io/tls   3      1m
ingress.networking.k8s.io/harbor-ingress     traefik   harbor.dvirlabs.com   192.168.10.240   80, 443   1m
```

### Check Certificate Details
```bash
kubectl describe certificate harbor-ingress -n dev-tools
```

Should show:
- `Status: True`
- `Message: Certificate is up to date and has not expired`
- `Issuer: letsencrypt` (ClusterIssuer)

### Test Access
```bash
curl -I https://harbor.dvirlabs.com
```

Should return `HTTP/2 200` (not 502)

### Verify Certificate in Browser
Visit `https://harbor.dvirlabs.com` - should show:
- Valid Let's Encrypt certificate
- Issued to: harbor.dvirlabs.com
- Issued by: Let's Encrypt Authority

---

## Alternative: Explicit Certificate Resource

If you prefer **declarative Certificate management** instead of ingress annotations, you can:

### Create `manifests/harbor/certificate.yaml`:
```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: harbor-ingress
  namespace: dev-tools
spec:
  secretName: harbor-ingress
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer
  dnsNames:
    - harbor.dvirlabs.com
```

### Remove ingress annotation:
In `manifests/harbor/values.yaml`, remove:
```yaml
cert-manager.io/cluster-issuer: letsencrypt  # Remove this line
```

### Why I recommend ingress annotations instead:
1. **Simpler**: No separate Certificate file to manage
2. **DRY principle**: Hostname defined once in values.yaml
3. **Less resources**: One less manifest file
4. **Harbor chart native**: Uses Harbor's standard annotation mechanism

---

## Production Checklist

- [x] Harbor chart uses `certSource: secret` (not "auto")
- [x] Secret name is stable: `harbor-ingress`
- [x] cert-manager annotation present on ingress
- [x] ClusterIssuer `letsencrypt` exists and is ready
- [x] Old self-signed secret deleted before redeployment
- [x] Traefik-specific annotations only (no nginx annotations)
- [x] HTTPS entrypoint configured: `websecure`
- [x] externalURL uses HTTPS: `https://harbor.dvirlabs.com`
- [x] GitOps workflow preserved (ArgoCD manages Harbor)
- [x] No duplicate resource names (no -v2, -copy suffixes)

---

## Troubleshooting

### Certificate stuck in "Issuing" state
```bash
# Check certificate details
kubectl describe certificate harbor-ingress -n dev-tools

# Check ACME order
kubectl get order -n dev-tools

# Check ACME challenge
kubectl get challenge -n dev-tools
```

### DNS-01 validation fails
```bash
# Check Cloudflare credentials secret
kubectl get secret cloudflare-api-token-secret -n cert-manager

# Check ClusterIssuer status
kubectl describe clusterissuer letsencrypt
```

### Harbor pods healthy but 502 error persists
```bash
# Check if secret exists and has valid cert
kubectl get secret harbor-ingress -n dev-tools -o yaml

# Restart Traefik to reload ingress config
kubectl rollout restart deployment traefik -n kube-system

# Check Traefik logs
kubectl logs -n kube-system -l app.kubernetes.io/name=traefik --tail=50
```

---

## Summary

**Before:**
- Harbor generates self-signed cert → `harbor-ingress` secret
- cert-manager tries to manage same secret → **IncorrectIssuer conflict**
- ACME orders fail repeatedly

**After:**
- cert-manager manages certificate lifecycle cleanly
- Harbor references the cert-manager-created secret
- Single source of truth for TLS management
- Clean GitOps workflow with stable resource names

**Files Changed:**
- `manifests/harbor/values.yaml` - TLS configuration fixed

**No Changes Needed:**
- Harbor chart templates (work as designed)
- Ingress class (traefik)
- ClusterIssuer (letsencrypt)
- Secret name (harbor-ingress)