rsyslog/README_GITOPS_STATUS.md

rsyslog GitOps: gitops-status-server Integration

Overview

This repository now uses gitops-status-server (a Kubernetes-based service) for GitOps status monitoring, replacing Prometheus Pushgateway.

Status: ✅ Ready for deployment

---

What This Means

For You (Operator)

• Configure Woodpecker cron job once (5 minutes)
• Cron runs every 2 minutes and checks if rsyslog config on server matches Git
• If changes detected, JSON is automatically sent to gitops-status-server
• Grafana shows:
  • Sync status (green=SYNCED, red=OUT_OF_SYNC)
  • Drift count (how many files changed)
  • Changed files (list of which files are different)
  • Last check (when drift was last checked)

For Grafana

• Connects to gitops-status-server via Infinity datasource
• Reads JSON status snapshots
• Displays rich metadata about rsyslog configuration state

For the System

• Detects manual edits on the server within 2 minutes
• Verifies deployment succeeded immediately after push
• No Pushgateway required
• Clean JSON-based architecture

---

Quick Start (5 Steps)

Step 1: Verify Files Are In Place

# Check these files exist:
ls -la .woodpecker.yml                                 # ✓ Updated
ls -la ansible/playbooks/drift-check.yml              # ✓ Enhanced
ls -la update-gitops-status.sh                        # ✓ New script

Step 2: Make Script Executable

chmod +x update-gitops-status.sh
git add update-gitops-status.sh
git commit -m "add: gitops-status-server integration script"
git push

Step 3: Verify Pipeline Runs

• Push should trigger Woodpecker pipeline
• New step update-gitops-status should execute after deploy
• Check logs for: "✓ Status update successful"

Step 4: Configure Woodpecker Cron Job

In Woodpecker UI:

1. Go to repository
2. Click Settings → Cron
3. Click Add Cron
4. Fill in:
   • Name: gitops_sync_check
   • Branch: master
   • Schedule: */2 * * * *
5. Click Save

Step 5: Test Cron Job

• Wait max 2 minutes for cron to trigger
• Check Woodpecker logs for cron execution
• Verify gitops-status-server received POST:

  kubectl logs -n observability-stack -l app=gitops-status-server | grep POST
  

---

How It Works

Architecture Diagram

Every 2 minutes (or after deployment):

Woodpecker
├─ gitops_sync_check or update-gitops-status step
└─ Runs: update-gitops-status.sh
   ├─ 1. Execute: ansible/playbooks/drift-check.yml
   │  └─ Compare Git files vs server files (read-only)
   │  └─ Output: DRIFTED_FILES=file1,file2,file3
   ├─ 2. Parse: Extract changed files and sync status
   ├─ 3. Generate: JSON payload with metadata
   └─ 4. POST: JSON to gitops-status-server/api/status

gitops-status-server (K8s Service)
└─ Receives JSON
└─ Updates /status.json
└─ Serves via HTTP

Grafana
└─ Infinity datasource
└─ Polls /status.json (periodic refresh)
└─ Displays in dashboard panel

Data Flow

Input: Ansible drift-check output

DRIFTED_FILES=/etc/rsyslog.conf,/etc/rsyslog.d/30-lab.conf
SYNC_STATUS=OUT_OF_SYNC

Output: JSON sent to gitops-status-server

{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "OUT_OF_SYNC",
  "drift_count": 2,
  "files": [
    { "name": "rsyslog.conf" },
    { "name": "rsyslog.d/30-lab.conf" }
  ],
  "last_check": "2026-04-21T10:30:00Z"
}

Display: Grafana dashboard

• 🔴 OUT_OF_SYNC (red card)
• Drift count: 2
• Files: rsyslog.conf, rsyslog.d/30-lab.conf
• Last check: 2026-04-21 10:30 UTC

---

Files Changed

.woodpecker.yml

Changes:

• Renamed step: update-sync-metric → update-gitops-status
• Changed command: replaced Pushgateway push → update-gitops-status.sh script call
• Added environment variables:
  • GITOPS_STATUS_SERVER_URL
  • REPO_NAME
  • SERVER_NAME
• Both update-gitops-status (post-deploy) and gitops_sync_check (cron) now use the script

ansible/playbooks/drift-check.yml

Changes:

• Added file collection: builds list of changed files in drifted_files fact
• Added debug output: prints DRIFTED_FILES=file1,file2,file3 for script parsing
• Added status markers: prints SYNC_STATUS=SYNCED or SYNC_STATUS=OUT_OF_SYNC
• No changes to drift detection logic (fully backward compatible)

update-gitops-status.sh (NEW)

Purpose: Orchestrates status generation and delivery 4 Steps:

1. Run drift-check.yml
2. Parse output to extract changed files
3. Build JSON payload
4. POST to gitops-status-server/api/status

---

Testing

Test 1: Verify script works locally

# From repo root, with SSH key configured
./update-gitops-status.sh

# Expected output:
# ═══════════════════════════════════════════════════
# Step 1/4: Running drift-check playbook...
# [playbook output...]
# Step 2/4: Analyzing drift detection results...
# Step 3/4: Building JSON payload...
# Generated JSON:
# {
#   "repo": "rsyslog",
#   ...
# }
# Step 4/4: Sending status to gitops-status-server...
# Response: HTTP 200
# ✓ Status update successful

Test 2: Verify Woodpecker pipeline runs

1. Make a change to a file
2. Push to master branch
3. Woodpecker pipeline should:
   • Run syntax-check ✓
   • Run validate ✓
   • Run deploy ✓
   • Run update-gitops-status ✓
4. Check logs for: "✓ Status update successful"

Test 3: Verify cron job triggers

1. Woodpecker cron job configured for */2 * * * *
2. Wait 2 minutes
3. Check Woodpecker UI for cron execution
4. Check logs for drift-check output

Test 4: Verify gitops-status-server receives JSON

# Check gitops-status-server logs
kubectl logs -n observability-stack -l app=gitops-status-server -f

# Should show POST requests:
# POST /api/status - 200 OK

Test 5: Verify Grafana dashboard

1. Open Grafana
2. Check Infinity datasource:
   • Should show "Data source is working"
3. Check dashboard panel:
   • Should display sync status
   • Should show drift count
   • Should list changed files (if any)

---

Troubleshooting

Issue: Cron job not running

Check:

1. Is cron job configured in Woodpecker?
   • Go to repo settings → Cron
   • Should see gitops_sync_check with */2 * * * * schedule
2. Is the schedule active?
   • Cron should have triggered at least once

Fix:

• Create cron job if missing
• Verify schedule is */2 * * * *
• Check branch is master

Issue: "HTTP 500" or "HTTP 503" when posting

Check:

1. Is gitops-status-server running?

   kubectl get pod -n observability-stack | grep gitops-status
   

2. Is it ready?

   kubectl get pod -n observability-stack -o wide | grep gitops-status
   

3. Check logs:

   kubectl logs -n observability-stack -l app=gitops-status-server
   

Fix:

• Restart gitops-status-server if needed
• Check error logs for API issues
• Verify /api/status endpoint exists

Issue: Drift not detected

Check:

1. Run drift-check manually:

   ansible-playbook -i ansible/inventory/hosts.yml ansible/playbooks/drift-check.yml -v
   

2. Does it report the correct status?
3. SSH connectivity to server?

Fix:

• Check SSH key is set in Woodpecker secrets
• Verify server files are readable: ssh user@server ls -la /etc/rsyslog.conf
• Check Ansible inventory is correct

Issue: JSON not sent (HTTP 000)

Check:

1. Is gitops-status-server URL correct?

   curl http://gitops-status-server.observability-stack.svc.cluster.local:80
   

2. Can Woodpecker reach it?
   • May be network/DNS issue

Fix:

• Check gitops-status-server hostname/port
• Test from Woodpecker container: curl http://gitops-status-server:80
• Check Woodpecker network policies

Issue: Grafana shows "No data"

Check:

1. Does Infinity datasource work?
   • Go to Data Sources → test
2. Can Grafana reach gitops-status-server?
3. What query is the panel using?

Fix:

• Verify datasource URL is correct
• Check query in panel: should be /status.json or similar
• Ensure gitops-status-server is returning JSON

---

Examples

Example 1: Server is synced

{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "SYNCED",
  "drift_count": 0,
  "files": [],
  "last_check": "2026-04-21T10:32:00Z"
}

Grafana display:

• 🟢 SYNCED
• Drift: 0
• Files: (empty)

Example 2: Manual edit on server

{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "OUT_OF_SYNC",
  "drift_count": 1,
  "files": [
    { "name": "rsyslog.conf" }
  ],
  "last_check": "2026-04-21T10:34:00Z"
}

Grafana display:

• 🔴 OUT OF SYNC
• Drift: 1
• Files: rsyslog.conf

Example 3: Multiple files changed after deployment

{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "SYNCED",
  "drift_count": 0,
  "files": [],
  "last_check": "2026-04-21T10:36:00Z"
}

Timeline:

1. 10:30 - Push to master triggers deploy
2. 10:31 - Deploy completes, files changed
3. 10:31 - update-gitops-status runs, verifies sync
4. 10:31 - JSON sent: SYNCED
5. 10:36 - Grafana shows ✓ SYNCED (5 min later)

---

Environment Configuration

The following environment variables are set in .woodpecker.yml:

GITOPS_STATUS_SERVER_URL: http://gitops-status-server.observability-stack.svc.cluster.local:80
REPO_NAME: rsyslog
SERVER_NAME: rsyslog-lab
SSH_PRIVATE_KEY: from_secret: SSH_PRIVATE_KEY
ANSIBLE_CONFIG: ansible.cfg

To customize:

• Edit .woodpecker.yml
• Change environment variables under update-gitops-status and gitops_sync_check steps
• Push and re-run

---

Key Features

✅ Automatic drift detection – Every 2 minutes
✅ Post-deployment verification – Immediate after deploy
✅ File-level details – Shows which files changed
✅ No Pushgateway – Simplified infrastructure
✅ Grafana integration – Infinity datasource (native)
✅ Audit trail – JSON snapshots with timestamps
✅ Multi-server ready – Structured for scale

---

Documentation

Document	Purpose
GITOPS_STATUS_SERVER_INTEGRATION.md	Comprehensive architecture & flow
QUICK_REFERENCE.md	Quick start & troubleshooting
REFACTOR_SUMMARY.md	Before/after comparison
This file	Overview & quick start

---

Next Steps

1. Verify files are in place:

   git status
   

2. Push changes:

   git push
   

3. Monitor first pipeline run:

   • Check Woodpecker logs
   • Look for update-gitops-status step
   • Verify HTTP 200 response

4. Configure cron job:

   • Go to Woodpecker UI
   • Add cron: gitops_sync_check at */2 * * * *

5. Test cron execution:

   • Wait 2 minutes
   • Check Woodpecker logs
   • Verify gitops-status-server receives JSON

6. Verify Grafana:

   • Check dashboard displays sync status
   • Test with manual file edit on server
   • Verify detection within 2 minutes

---

Support

For issues or questions:

1. Check QUICK_REFERENCE.md troubleshooting section
2. Review Woodpecker pipeline logs
3. Check gitops-status-server application logs:

   kubectl logs -n observability-stack -l app=gitops-status-server -f
   

4. Test connectivity manually:

   curl http://gitops-status-server.observability-stack.svc.cluster.local:80/api/status
   

---

Summary

You now have a clean, production-ready GitOps status monitoring system that:

• Detects configuration drift every 2 minutes
• Sends rich metadata (file names, timestamps) to gitops-status-server
• Integrates with Grafana via Infinity datasource
• Requires minimal infrastructure (no Pushgateway)
• Works reliably for multi-server deployments

Status: ✅ Ready to deploy and use