228 lines
6.6 KiB
Markdown
228 lines
6.6 KiB
Markdown
# Quick Reference: GitOps Status Server Implementation
|
||
|
||
## What Changed
|
||
|
||
✅ **Removed:** Pushgateway-based metrics push for sync status
|
||
✅ **Added:** JSON-based status updates to gitops-status-server
|
||
✅ **Kept:** All existing deploy/apply/drift-check logic
|
||
|
||
## Files Modified/Created
|
||
|
||
### Modified
|
||
- **`.woodpecker.yml`** – Updated to use `update-gitops-status.sh` instead of Pushgateway
|
||
- **`ansible/playbooks/drift-check.yml`** – Added structured file output (`DRIFTED_FILES=...`)
|
||
|
||
### Created/Used
|
||
- **`update-gitops-status.sh`** – Main script that generates JSON and POSTs to gitops-status-server
|
||
- **`GITOPS_STATUS_SERVER_INTEGRATION.md`** – Full documentation
|
||
|
||
## How It Works
|
||
|
||
```
|
||
Cron (every 2 min) or Post-Deploy
|
||
↓
|
||
update-gitops-status.sh
|
||
1. Run drift-check.yml
|
||
2. Parse output (DRIFTED_FILES=...)
|
||
3. Generate JSON with metadata
|
||
4. POST to gitops-status-server/api/status
|
||
↓
|
||
gitops-status-server receives JSON
|
||
└─ Updates /status.json internally
|
||
↓
|
||
Grafana Infinity datasource
|
||
└─ Queries /status.json
|
||
└─ Displays sync status, drift count, changed files
|
||
```
|
||
|
||
## JSON Payload Example
|
||
|
||
### When Synced:
|
||
```json
|
||
{
|
||
"repo": "rsyslog",
|
||
"server": "rsyslog-lab",
|
||
"sync_status": "SYNCED",
|
||
"drift_count": 0,
|
||
"files": [],
|
||
"last_check": "2026-04-21T10:30:00Z"
|
||
}
|
||
```
|
||
|
||
### When Out of Sync:
|
||
```json
|
||
{
|
||
"repo": "rsyslog",
|
||
"server": "rsyslog-lab",
|
||
"sync_status": "OUT_OF_SYNC",
|
||
"drift_count": 2,
|
||
"files": [
|
||
{ "name": "rsyslog.conf" },
|
||
{ "name": "rsyslog.d/30-lab.conf" }
|
||
],
|
||
"last_check": "2026-04-21T10:30:00Z"
|
||
}
|
||
```
|
||
|
||
## Testing
|
||
|
||
### Test the script locally:
|
||
```bash
|
||
# Make script executable
|
||
chmod +x update-gitops-status.sh
|
||
|
||
# Run it (requires SSH key and Ansible configured)
|
||
./update-gitops-status.sh
|
||
|
||
# You should see:
|
||
# ==> Running drift-check playbook...
|
||
# Step 1/4: Running drift-check playbook...
|
||
# Step 2/4: Analyzing drift detection results...
|
||
# Step 3/4: Building JSON payload...
|
||
# Step 4/4: Sending status to gitops-status-server...
|
||
```
|
||
|
||
### Test drift-check.yml output:
|
||
```bash
|
||
# Run drift-check playbook to see new structured output
|
||
ansible-playbook -i ansible/inventory/hosts.yml ansible/playbooks/drift-check.yml
|
||
|
||
# You should see debug output like:
|
||
# DRIFTED_FILES=/etc/rsyslog.conf,/etc/rsyslog.d/30-lab.conf
|
||
# SYNC_STATUS=OUT_OF_SYNC
|
||
```
|
||
|
||
### Test gitops-status-server connectivity:
|
||
```bash
|
||
# From Woodpecker container or CI environment, test endpoint
|
||
curl -X POST \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"repo":"rsyslog","server":"rsyslog-lab","sync_status":"SYNCED","drift_count":0,"files":[],"last_check":"2026-04-21T10:30:00Z"}' \
|
||
http://gitops-status-server.observability-stack.svc.cluster.local:80/api/status
|
||
```
|
||
|
||
## Deployment Steps
|
||
|
||
### 1. Push changes to Git
|
||
```bash
|
||
git add .woodpecker.yml ansible/playbooks/drift-check.yml update-gitops-status.sh
|
||
git commit -m "refactor: replace pushgateway with gitops-status-server integration"
|
||
git push
|
||
```
|
||
|
||
### 2. Verify Woodpecker pipeline
|
||
- Pipeline should run automatically on push
|
||
- Check logs for successful syntax-check, validate, deploy steps
|
||
- New post-deploy step `update-gitops-status` should run after deploy
|
||
|
||
### 3. Set up Woodpecker cron job
|
||
In Woodpecker UI:
|
||
1. Go to repository settings
|
||
2. Add Cron job:
|
||
- Name: `gitops_sync_check`
|
||
- Branch: `master`
|
||
- Schedule: `*/2 * * * *` (every 2 minutes)
|
||
|
||
### 4. Verify cron execution
|
||
- Wait for next cron trigger (within 2 minutes)
|
||
- Check Woodpecker logs for cron execution
|
||
- Look for: "Step 1/4: Running drift-check playbook..."
|
||
- Should show: "✓ Status update successful (HTTP 200)"
|
||
|
||
### 5. Verify gitops-status-server receives JSON
|
||
```bash
|
||
# Check gitops-status-server logs
|
||
kubectl logs -n observability-stack -l app=gitops-status-server -f
|
||
|
||
# Should show POST requests like:
|
||
# POST /api/status from Woodpecker
|
||
```
|
||
|
||
### 6. Verify Grafana dashboard
|
||
- Open Grafana
|
||
- Check Infinity datasource can query gitops-status-server
|
||
- Dashboard panel should display:
|
||
- Sync status (SYNCED / OUT_OF_SYNC)
|
||
- Drift count
|
||
- List of changed files
|
||
- Last check timestamp
|
||
|
||
## Environment Variables
|
||
|
||
In `.woodpecker.yml`:
|
||
```yaml
|
||
environment:
|
||
GITOPS_STATUS_SERVER_URL: http://gitops-status-server.observability-stack.svc.cluster.local:80
|
||
REPO_NAME: rsyslog
|
||
SERVER_NAME: rsyslog-lab
|
||
SSH_PRIVATE_KEY: from_secret: SSH_PRIVATE_KEY
|
||
ANSIBLE_CONFIG: ansible.cfg
|
||
```
|
||
|
||
## Troubleshooting
|
||
|
||
### "HTTP 500" when posting to gitops-status-server
|
||
- Check gitops-status-server logs:
|
||
`kubectl logs -n observability-stack -l app=gitops-status-server`
|
||
- Verify gitops-status-server's API expects POST /api/status
|
||
- Check JSON format matches expected schema
|
||
|
||
### Cron job not running
|
||
- In Woodpecker UI, check cron job list in repository settings
|
||
- Verify schedule is `*/2 * * * *`
|
||
- Check repository has write access to cron jobs
|
||
|
||
### Drift not detected
|
||
- Run drift-check.yml manually:
|
||
`ansible-playbook -i ansible/inventory/hosts.yml ansible/playbooks/drift-check.yml -v`
|
||
- Check SSH key is properly set in Woodpecker secrets
|
||
- Verify server files are readable via SSH
|
||
|
||
### JSON not being sent
|
||
- Check update-gitops-status.sh script is executable:
|
||
`ls -la update-gitops-status.sh`
|
||
- Check Woodpecker logs for HTTP response code
|
||
- Verify gitops-status-server URL is correct:
|
||
`curl http://gitops-status-server.observability-stack.svc.cluster.local:80`
|
||
|
||
## Key Differences from Previous Architecture
|
||
|
||
| Old (Pushgateway) | New (gitops-status-server) |
|
||
|-------------------|---------------------------|
|
||
| POST metric to Pushgateway | POST JSON to gitops-status-server |
|
||
| Only sync/out-of-sync (0/1) | Rich JSON with file names, count, timestamp |
|
||
| Prometheus dependency | Pure JSON API |
|
||
| Pushgateway metric format | Grafana Infinity datasource |
|
||
| Manual file-level details | Automatic file list in JSON |
|
||
|
||
## Success Indicators
|
||
|
||
After deployment:
|
||
- ✓ Cron job runs every 2 minutes
|
||
- ✓ update-gitops-status.sh script executes
|
||
- ✓ JSON POST to gitops-status-server returns HTTP 200
|
||
- ✓ gitops-status-server logs show POST requests
|
||
- ✓ Grafana dashboard displays sync status and file names
|
||
- ✓ Manual changes on server detected within 2 minutes
|
||
- ✓ Deployment status updated immediately after push
|
||
|
||
## Rollback
|
||
|
||
If needed, revert to Pushgateway:
|
||
```bash
|
||
# Checkout old .woodpecker.yml version
|
||
git checkout HEAD~N .woodpecker.yml
|
||
|
||
# (Where N is number of commits back)
|
||
# Or manually restore Pushgateway step
|
||
```
|
||
|
||
## Monitoring
|
||
|
||
After deployment, monitor:
|
||
1. Cron execution frequency (should be every 2 minutes)
|
||
2. HTTP response codes (should be 200)
|
||
3. JSON schema consistency
|
||
4. Grafana dashboard updates
|
||
5. Time to drift detection (should be ≤ 2 minutes)
|