rsyslog/DEPLOYMENT_CHECKLIST.md

# Deployment Checklist

This checklist guides you through deploying the updated rsyslog repository with gitops-status-server integration.

## Prerequisites

Before deploying, ensure:

- [ ] gitops-status-server is deployed and accessible at:  
      `http://gitops-status-server.observability-stack.svc.cluster.local:80`
- [ ] gitops-status-server has `/api/status` endpoint implemented (see [GITOPS_STATUS_API_REFERENCE.md](GITOPS_STATUS_API_REFERENCE.md))
- [ ] Woodpecker CI is configured for this repository
- [ ] SSH access to rsyslog-lab server is configured
- [ ] SSH_PRIVATE_KEY secret is set in Woodpecker repository settings

## Files Changed/Added

### New Files
- ✅ `update-gitops-status.sh` - Main script for JSON status generation
- ✅ `GITOPS_STATUS_INTEGRATION.md` - Integration documentation
- ✅ `GITOPS_STATUS_API_REFERENCE.md` - API reference with examples
- ✅ `MIGRATION_SUMMARY.md` - Summary of all changes made

### Modified Files
- ✅ `.woodpecker.yml` - Updated pipeline to use JSON status
- ✅ `README.md` - Updated documentation with new flow

### Unchanged Files (no action needed)
- ✅ All Ansible playbooks
- ✅ All Ansible inventory files
- ✅ All rsyslog config files
- ✅ Local scripts (apply.sh, drift-check.sh)

## Deployment Steps

### 1. Review Changes

- [ ] Read [MIGRATION_SUMMARY.md](MIGRATION_SUMMARY.md) to understand all changes
- [ ] Review [.woodpecker.yml](.woodpecker.yml) pipeline changes
- [ ] Review [update-gitops-status.sh](update-gitops-status.sh) script logic

### 2. Verify gitops-status-server

Test the gitops-status-server API endpoint:

```bash
# Test POST endpoint (should return 200 or 404 if not implemented yet)
curl -X POST http://gitops-status-server.observability-stack.svc.cluster.local:80/api/status \
  -H "Content-Type: application/json" \
  -d '{
    "repo": "test",
    "server": "test",
    "sync_status": "SYNCED",
    "drift_count": 0,
    "files": [],
    "last_check": "2026-04-21T10:00:00Z"
  }'

# Test GET endpoint
curl http://gitops-status-server.observability-stack.svc.cluster.local:80/status.json
```

If the API is not implemented yet:
- [ ] Implement gitops-status-server API (use [GITOPS_STATUS_API_REFERENCE.md](GITOPS_STATUS_API_REFERENCE.md))
- [ ] Deploy to Kubernetes cluster
- [ ] Verify endpoints are accessible

### 3. Test Locally (Optional)

Before pushing to Git, you can test the script locally:

```bash
# Set environment variables
export GITOPS_STATUS_SERVER_URL="http://gitops-status-server.observability-stack.svc.cluster.local:80"
export REPO_NAME="rsyslog"
export SERVER_NAME="rsyslog-lab"

# Make script executable
chmod +x update-gitops-status.sh

# Run the script (requires Ansible, jq, curl)
./update-gitops-status.sh
```

Expected output:
```
==> Running drift check playbook...
    Inventory: ansible/inventory/hosts.yml
    Playbook:  ansible/playbooks/drift-check.yml
==> Status: SYNCED - server configuration matches Git
==> Drift count: 0
==> Generated JSON status:
{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "SYNCED",
  "drift_count": 0,
  "files": [],
  "last_check": "2026-04-21T10:30:00Z"
}
==> Sending status to gitops-status-server...
    URL: http://gitops-status-server.observability-stack.svc.cluster.local:80/api/status
==> Status update successful (HTTP 200)
```

### 4. Commit and Push Changes

```bash
# Stage all changes
git add .

# Commit
git commit -m "Migrate from Pushgateway to gitops-status-server JSON status

- Add update-gitops-status.sh script for JSON status generation
- Update .woodpecker.yml to use gitops-status-server
- Remove Pushgateway metric push logic
- Add comprehensive documentation
- Keep all Ansible playbooks unchanged"

# Push to master (will trigger deploy pipeline)
git push origin master
```

### 5. Monitor First Deployment

After pushing to master:

- [ ] Watch Woodpecker pipeline execution
- [ ] Verify `syntax-check` step passes
- [ ] Verify `validate` step passes
- [ ] Verify `deploy` step completes
- [ ] Verify `update-gitops-status` step runs successfully
- [ ] Check that JSON is sent to gitops-status-server

Example successful `update-gitops-status` step output:
```
==> Running post-deploy GitOps status check...
==> Running drift check playbook...
==> Status: SYNCED - server configuration matches Git
==> Drift count: 0
==> Generated JSON status: {...}
==> Sending status to gitops-status-server...
==> Status update successful (HTTP 200)
==> JSON status update complete. Pipeline always succeeds.
```

### 6. Verify Cron Pipeline

The cron pipeline runs every 2 minutes:

- [ ] Wait for next cron execution
- [ ] Check Woodpecker for `gitops_sync_check` pipeline run
- [ ] Verify JSON status is sent
- [ ] Verify pipeline succeeds (if synced) or fails (if drift detected)

### 7. Test Drift Detection

Manually create drift to test detection:

```bash
# SSH to the server
ssh rsyslog-lab

# Edit a config file
echo "# manual edit" >> /etc/rsyslog.conf

# Wait up to 2 minutes for next cron run
```

Expected behavior:
- [ ] Cron pipeline runs
- [ ] Drift detected in Ansible playbook
- [ ] JSON sent with `sync_status: "OUT_OF_SYNC"`
- [ ] JSON includes `files: [{"name": "rsyslog.conf"}]`
- [ ] Pipeline marked as FAILED (for visibility)

Verify in gitops-status-server:
```bash
curl http://gitops-status-server.observability-stack.svc.cluster.local:80/status.json
```

Should show:
```json
[
  {
    "repo": "rsyslog",
    "server": "rsyslog-lab",
    "sync_status": "OUT_OF_SYNC",
    "drift_count": 1,
    "files": [
      {"name": "rsyslog.conf"}
    ],
    "last_check": "...",
    "updated_at": "..."
  }
]
```

### 8. Configure Grafana Dashboard

- [ ] Add Infinity datasource pointing to gitops-status-server
- [ ] Create dashboard to display GitOps status
- [ ] Add panels for:
  - Sync status overview (SYNCED vs OUT_OF_SYNC)
  - Drift count per repo
  - Detailed file list for drifted repos
  - Last check timestamp
  - Historical trend (if gitops-status-server stores history)

Example dashboard queries provided in [GITOPS_STATUS_API_REFERENCE.md](GITOPS_STATUS_API_REFERENCE.md).

### 9. Cleanup (Optional)

If everything works correctly:

- [ ] Remove old Pushgateway metrics for rsyslog (if no longer needed)
- [ ] Update any alerts/dashboards that used old `gitops_sync_status` metric
- [ ] Document the new JSON status format in team wiki/docs

## Troubleshooting

### Script fails with "command not found: jq"

**Problem:** `jq` is not installed in the Woodpecker container

**Solution:** The `.woodpecker.yml` already includes `apk add --no-cache jq`. Verify the step runs before the script.

### Script fails with "HTTP 404" or "HTTP 500"

**Problem:** gitops-status-server endpoint not implemented or not accessible

**Solution:**
1. Verify gitops-status-server is running: `curl http://gitops-status-server.observability-stack.svc.cluster.local:80/health`
2. Check Kubernetes service: `kubectl get svc -n observability-stack`
3. Implement `/api/status` endpoint using [GITOPS_STATUS_API_REFERENCE.md](GITOPS_STATUS_API_REFERENCE.md)

### Script fails with "No such file or directory: update-gitops-status.sh"

**Problem:** Script not found in workspace

**Solution:** The script is created in the repository root. Verify it's committed to Git and available in the CI workspace.

### Drift not detected when expected

**Problem:** Manual changes not showing up in drift check

**Solution:**
1. Verify changes are to files managed by Git (rsyslog.conf or rsyslog.d/*.conf)
2. Check Ansible playbook output for diff details
3. Verify SSH access to server from CI container

### JSON has empty files array even when drift detected

**Problem:** Script parsing logic not extracting filenames correctly

**Solution:**
1. Check Ansible output format - script expects specific JSON structure
2. Run with `ANSIBLE_STDOUT_CALLBACK=json` to see raw output
3. Update regex patterns in `update-gitops-status.sh` if needed

## Rollback Procedure

If you need to rollback to Pushgateway:

```bash
# Revert to previous commit
git revert HEAD

# Or restore specific files
git checkout HEAD~1 .woodpecker.yml
git checkout HEAD~1 README.md

# Remove new files
git rm update-gitops-status.sh GITOPS_STATUS_*.md MIGRATION_SUMMARY.md

# Commit and push
git commit -m "Rollback to Pushgateway metrics"
git push origin master
```

## Success Criteria

✅ All checks passed when:

- [ ] Woodpecker pipeline completes successfully on push to master
- [ ] JSON status is sent to gitops-status-server after deploy
- [ ] Cron pipeline runs every 2 minutes and sends JSON status
- [ ] Drift is correctly detected and reported in JSON
- [ ] gitops-status-server `/status.json` endpoint returns correct data
- [ ] Grafana dashboard displays rsyslog sync status
- [ ] No errors in Woodpecker logs
- [ ] File-level drift details are visible in Grafana

## Additional Resources

- [README.md](README.md) - Repository overview and workflow
- [MIGRATION_SUMMARY.md](MIGRATION_SUMMARY.md) - Detailed migration changes
- [GITOPS_STATUS_INTEGRATION.md](GITOPS_STATUS_INTEGRATION.md) - Integration architecture
- [GITOPS_STATUS_API_REFERENCE.md](GITOPS_STATUS_API_REFERENCE.md) - API implementation guide

## Support

If you encounter issues:

1. Check Woodpecker pipeline logs
2. Verify gitops-status-server logs
3. Test API endpoints manually with curl
4. Review Ansible playbook output
5. Check this repository's documentation files