# Quick Reference: GitOps Status Server Implementation

## What Changed

✅ **Removed:** Pushgateway-based metrics push for sync status  
✅ **Added:** JSON-based status updates to gitops-status-server  
✅ **Kept:** All existing deploy/apply/drift-check logic  

## Files Modified/Created

### Modified
- **`.woodpecker.yml`** – Updated to use `update-gitops-status.sh` instead of Pushgateway
- **`ansible/playbooks/drift-check.yml`** – Added structured file output (`DRIFTED_FILES=...`)

### Created/Used
- **`update-gitops-status.sh`** – Main script that generates JSON and POSTs to gitops-status-server
- **`GITOPS_STATUS_SERVER_INTEGRATION.md`** – Full documentation

## How It Works

```
Cron (every 2 min) or Post-Deploy
        ↓
update-gitops-status.sh
  1. Run drift-check.yml
  2. Parse output (DRIFTED_FILES=...)
  3. Generate JSON with metadata
  4. POST to gitops-status-server/api/status
        ↓
gitops-status-server receives JSON
  └─ Updates /status.json internally
        ↓
Grafana Infinity datasource
  └─ Queries /status.json
  └─ Displays sync status, drift count, changed files
```

## JSON Payload Example

### When Synced:
```json
{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "SYNCED",
  "drift_count": 0,
  "files": [],
  "last_check": "2026-04-21T10:30:00Z"
}
```

### When Out of Sync:
```json
{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "OUT_OF_SYNC",
  "drift_count": 2,
  "files": [
    { "name": "rsyslog.conf" },
    { "name": "rsyslog.d/30-lab.conf" }
  ],
  "last_check": "2026-04-21T10:30:00Z"
}
```

## Testing

### Test the script locally:
```bash
# Make script executable
chmod +x update-gitops-status.sh

# Run it (requires SSH key and Ansible configured)
./update-gitops-status.sh

# You should see:
# ==> Running drift-check playbook...
# Step 1/4: Running drift-check playbook...
# Step 2/4: Analyzing drift detection results...
# Step 3/4: Building JSON payload...
# Step 4/4: Sending status to gitops-status-server...
```

### Test drift-check.yml output:
```bash
# Run drift-check playbook to see new structured output
ansible-playbook -i ansible/inventory/hosts.yml ansible/playbooks/drift-check.yml

# You should see debug output like:
# DRIFTED_FILES=/etc/rsyslog.conf,/etc/rsyslog.d/30-lab.conf
# SYNC_STATUS=OUT_OF_SYNC
```

### Test gitops-status-server connectivity:
```bash
# From Woodpecker container or CI environment, test endpoint
curl -X POST \
  -H "Content-Type: application/json" \
  -d '{"repo":"rsyslog","server":"rsyslog-lab","sync_status":"SYNCED","drift_count":0,"files":[],"last_check":"2026-04-21T10:30:00Z"}' \
  http://gitops-status-server.observability-stack.svc.cluster.local:80/api/status
```

## Deployment Steps

### 1. Push changes to Git
```bash
git add .woodpecker.yml ansible/playbooks/drift-check.yml update-gitops-status.sh
git commit -m "refactor: replace pushgateway with gitops-status-server integration"
git push
```

### 2. Verify Woodpecker pipeline
- Pipeline should run automatically on push
- Check logs for successful syntax-check, validate, deploy steps
- New post-deploy step `update-gitops-status` should run after deploy

### 3. Set up Woodpecker cron job
In Woodpecker UI:
1. Go to repository settings
2. Add Cron job:
   - Name: `gitops_sync_check`
   - Branch: `master`
   - Schedule: `*/2 * * * *` (every 2 minutes)

### 4. Verify cron execution
- Wait for next cron trigger (within 2 minutes)
- Check Woodpecker logs for cron execution
- Look for: "Step 1/4: Running drift-check playbook..."
- Should show: "✓ Status update successful (HTTP 200)"

### 5. Verify gitops-status-server receives JSON
```bash
# Check gitops-status-server logs
kubectl logs -n observability-stack -l app=gitops-status-server -f

# Should show POST requests like:
# POST /api/status from Woodpecker
```

### 6. Verify Grafana dashboard
- Open Grafana
- Check Infinity datasource can query gitops-status-server
- Dashboard panel should display:
  - Sync status (SYNCED / OUT_OF_SYNC)
  - Drift count
  - List of changed files
  - Last check timestamp

## Environment Variables

In `.woodpecker.yml`:
```yaml
environment:
  GITOPS_STATUS_SERVER_URL: http://gitops-status-server.observability-stack.svc.cluster.local:80
  REPO_NAME: rsyslog
  SERVER_NAME: rsyslog-lab
  SSH_PRIVATE_KEY: from_secret: SSH_PRIVATE_KEY
  ANSIBLE_CONFIG: ansible.cfg
```

## Troubleshooting

### "HTTP 500" when posting to gitops-status-server
- Check gitops-status-server logs:  
  `kubectl logs -n observability-stack -l app=gitops-status-server`
- Verify gitops-status-server's API expects POST /api/status
- Check JSON format matches expected schema

### Cron job not running
- In Woodpecker UI, check cron job list in repository settings
- Verify schedule is `*/2 * * * *`
- Check repository has write access to cron jobs

### Drift not detected
- Run drift-check.yml manually:  
  `ansible-playbook -i ansible/inventory/hosts.yml ansible/playbooks/drift-check.yml -v`
- Check SSH key is properly set in Woodpecker secrets
- Verify server files are readable via SSH

### JSON not being sent
- Check update-gitops-status.sh script is executable:  
  `ls -la update-gitops-status.sh`
- Check Woodpecker logs for HTTP response code
- Verify gitops-status-server URL is correct:  
  `curl http://gitops-status-server.observability-stack.svc.cluster.local:80`

## Key Differences from Previous Architecture

| Old (Pushgateway) | New (gitops-status-server) |
|-------------------|---------------------------|
| POST metric to Pushgateway | POST JSON to gitops-status-server |
| Only sync/out-of-sync (0/1) | Rich JSON with file names, count, timestamp |
| Prometheus dependency | Pure JSON API |
| Pushgateway metric format | Grafana Infinity datasource |
| Manual file-level details | Automatic file list in JSON |

## Success Indicators

After deployment:
- ✓ Cron job runs every 2 minutes
- ✓ update-gitops-status.sh script executes
- ✓ JSON POST to gitops-status-server returns HTTP 200
- ✓ gitops-status-server logs show POST requests
- ✓ Grafana dashboard displays sync status and file names
- ✓ Manual changes on server detected within 2 minutes
- ✓ Deployment status updated immediately after push

## Rollback

If needed, revert to Pushgateway:
```bash
# Checkout old .woodpecker.yml version
git checkout HEAD~N .woodpecker.yml

# (Where N is number of commits back)
# Or manually restore Pushgateway step
```

## Monitoring

After deployment, monitor:
1. Cron execution frequency (should be every 2 minutes)
2. HTTP response codes (should be 200)
3. JSON schema consistency
4. Grafana dashboard updates
5. Time to drift detection (should be ≤ 2 minutes)