rsyslog/QUICK_REFERENCE.md

Quick Reference: GitOps Status Server Implementation

What Changed

✅ Removed: Pushgateway-based metrics push for sync status
✅ Added: JSON-based status updates to gitops-status-server
✅ Kept: All existing deploy/apply/drift-check logic

Files Modified/Created

Modified

• .woodpecker.yml – Updated to use update-gitops-status.sh instead of Pushgateway
• ansible/playbooks/drift-check.yml – Added structured file output (DRIFTED_FILES=...)

Created/Used

• update-gitops-status.sh – Main script that generates JSON and POSTs to gitops-status-server
• GITOPS_STATUS_SERVER_INTEGRATION.md – Full documentation

How It Works

Cron (every 2 min) or Post-Deploy
        ↓
update-gitops-status.sh
  1. Run drift-check.yml
  2. Parse output (DRIFTED_FILES=...)
  3. Generate JSON with metadata
  4. POST to gitops-status-server/api/status
        ↓
gitops-status-server receives JSON
  └─ Updates /status.json internally
        ↓
Grafana Infinity datasource
  └─ Queries /status.json
  └─ Displays sync status, drift count, changed files

JSON Payload Example

When Synced:

{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "SYNCED",
  "drift_count": 0,
  "files": [],
  "last_check": "2026-04-21T10:30:00Z"
}

When Out of Sync:

{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "OUT_OF_SYNC",
  "drift_count": 2,
  "files": [
    { "name": "rsyslog.conf" },
    { "name": "rsyslog.d/30-lab.conf" }
  ],
  "last_check": "2026-04-21T10:30:00Z"
}

Testing

Test the script locally:

# Make script executable
chmod +x update-gitops-status.sh

# Run it (requires SSH key and Ansible configured)
./update-gitops-status.sh

# You should see:
# ==> Running drift-check playbook...
# Step 1/4: Running drift-check playbook...
# Step 2/4: Analyzing drift detection results...
# Step 3/4: Building JSON payload...
# Step 4/4: Sending status to gitops-status-server...

Test drift-check.yml output:

# Run drift-check playbook to see new structured output
ansible-playbook -i ansible/inventory/hosts.yml ansible/playbooks/drift-check.yml

# You should see debug output like:
# DRIFTED_FILES=/etc/rsyslog.conf,/etc/rsyslog.d/30-lab.conf
# SYNC_STATUS=OUT_OF_SYNC

Test gitops-status-server connectivity:

# From Woodpecker container or CI environment, test endpoint
curl -X POST \
  -H "Content-Type: application/json" \
  -d '{"repo":"rsyslog","server":"rsyslog-lab","sync_status":"SYNCED","drift_count":0,"files":[],"last_check":"2026-04-21T10:30:00Z"}' \
  http://gitops-status-server.observability-stack.svc.cluster.local:80/api/status

Deployment Steps

1. Push changes to Git

git add .woodpecker.yml ansible/playbooks/drift-check.yml update-gitops-status.sh
git commit -m "refactor: replace pushgateway with gitops-status-server integration"
git push

2. Verify Woodpecker pipeline

• Pipeline should run automatically on push
• Check logs for successful syntax-check, validate, deploy steps
• New post-deploy step update-gitops-status should run after deploy

3. Set up Woodpecker cron job

In Woodpecker UI:

1. Go to repository settings
2. Add Cron job:
   • Name: gitops_sync_check
   • Branch: master
   • Schedule: */2 * * * * (every 2 minutes)

4. Verify cron execution

• Wait for next cron trigger (within 2 minutes)
• Check Woodpecker logs for cron execution
• Look for: "Step 1/4: Running drift-check playbook..."
• Should show: "✓ Status update successful (HTTP 200)"

5. Verify gitops-status-server receives JSON

# Check gitops-status-server logs
kubectl logs -n observability-stack -l app=gitops-status-server -f

# Should show POST requests like:
# POST /api/status from Woodpecker

6. Verify Grafana dashboard

• Open Grafana
• Check Infinity datasource can query gitops-status-server
• Dashboard panel should display:
  • Sync status (SYNCED / OUT_OF_SYNC)
  • Drift count
  • List of changed files
  • Last check timestamp

Environment Variables

In .woodpecker.yml:

environment:
  GITOPS_STATUS_SERVER_URL: http://gitops-status-server.observability-stack.svc.cluster.local:80
  REPO_NAME: rsyslog
  SERVER_NAME: rsyslog-lab
  SSH_PRIVATE_KEY: from_secret: SSH_PRIVATE_KEY
  ANSIBLE_CONFIG: ansible.cfg

Troubleshooting

"HTTP 500" when posting to gitops-status-server

• Check gitops-status-server logs:
  kubectl logs -n observability-stack -l app=gitops-status-server
• Verify gitops-status-server's API expects POST /api/status
• Check JSON format matches expected schema

Cron job not running

• In Woodpecker UI, check cron job list in repository settings
• Verify schedule is */2 * * * *
• Check repository has write access to cron jobs

Drift not detected

• Run drift-check.yml manually:
  ansible-playbook -i ansible/inventory/hosts.yml ansible/playbooks/drift-check.yml -v
• Check SSH key is properly set in Woodpecker secrets
• Verify server files are readable via SSH

JSON not being sent

• Check update-gitops-status.sh script is executable:
  ls -la update-gitops-status.sh
• Check Woodpecker logs for HTTP response code
• Verify gitops-status-server URL is correct:
  curl http://gitops-status-server.observability-stack.svc.cluster.local:80

Key Differences from Previous Architecture

Old (Pushgateway)	New (gitops-status-server)
POST metric to Pushgateway	POST JSON to gitops-status-server
Only sync/out-of-sync (0/1)	Rich JSON with file names, count, timestamp
Prometheus dependency	Pure JSON API
Pushgateway metric format	Grafana Infinity datasource
Manual file-level details	Automatic file list in JSON

Success Indicators

After deployment:

• ✓ Cron job runs every 2 minutes
• ✓ update-gitops-status.sh script executes
• ✓ JSON POST to gitops-status-server returns HTTP 200
• ✓ gitops-status-server logs show POST requests
• ✓ Grafana dashboard displays sync status and file names
• ✓ Manual changes on server detected within 2 minutes
• ✓ Deployment status updated immediately after push

Rollback

If needed, revert to Pushgateway:

# Checkout old .woodpecker.yml version
git checkout HEAD~N .woodpecker.yml

# (Where N is number of commits back)
# Or manually restore Pushgateway step

Monitoring

After deployment, monitor:

1. Cron execution frequency (should be every 2 minutes)
2. HTTP response codes (should be 200)
3. JSON schema consistency
4. Grafana dashboard updates
5. Time to drift detection (should be ≤ 2 minutes)