rsyslog/GITOPS_STATUS_SERVER_INTEGRATION.md

GitOps Status Server Integration - rsyslog Workflow

Overview

This document describes the rsyslog repository's integration with gitops-status-server, a centralized Kubernetes-based service that manages GitOps status information for multiple repositories.

Architecture: Rsyslog repo → generates JSON → POST to gitops-status-server → Grafana reads via Infinity datasource

---

Architecture

┌─ Scheduled Cron (every 2 minutes) ─────────────────────────────┐
│                                                                  │
│  Woodpecker Trigger: event: cron                               │
│        ↓                                                        │
│  Step: gitops_sync_check                                       │
│  └─ Run: update-gitops-status.sh                              │
│     ├─ Execute: ansible/playbooks/drift-check.yml             │
│     │  └─ Check rsyslog.conf and rsyslog.d/* files           │
│     │  └─ Detect drift vs Git repo                             │
│     │  └─ Output: DRIFTED_FILES=file1,file2,file3            │
│     ├─ Parse: Extract sync status and changed files          │
│     ├─ Generate: JSON payload with metadata                   │
│     └─ POST: JSON to gitops-status-server/api/status         │
│        ↓                                                        │
│  gitops-status-server (Kubernetes service)                     │
│  └─ Receives JSON                                              │
│  └─ Updates internal state                                     │
│  └─ Serves at: /status.json                                   │
│        ↓                                                        │
│  Grafana                                                        │
│  └─ Infinity datasource polls /status.json                    │
│  └─ Displays: sync status, drift count, changed files        │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘

┌─ Post-Deployment Verification (push to master) ─────────────────┐
│                                                                  │
│  Woodpecker Pipeline Event: push                               │
│  Triggered on: push to master branch                           │
│        ↓                                                        │
│  Steps: syntax-check → validate → deploy → update-gitops-status│
│                                                  ↓               │
│  Run: update-gitops-status.sh (same as above)                 │
│  └─ Verify deployment succeeded                               │
│  └─ Generate JSON status                                       │
│  └─ POST to gitops-status-server                              │
│        ↓                                                        │
│  gitops-status-server updates /status.json                     │
│        ↓                                                        │
│  Grafana reflects latest deployment status                     │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘

---

Components

1. Ansible Playbook: ansible/playbooks/drift-check.yml

Purpose: Detect configuration drift between Git repository and live server

Behavior:

• Runs in check mode (read-only, no actual changes)
• Uses Ansible copy module with check_mode: true
• Compares controller files (from Git) with server files
• Detects:
  • Changes in /etc/rsyslog.conf
  • Changes in /etc/rsyslog.d/*.conf files
  • Missing files on server

Output (debug messages):

• DRIFTED_FILES=file1,file2,file3 – Comma-separated list of changed files
• SYNC_STATUS=SYNCED or SYNC_STATUS=OUT_OF_SYNC – Simple status indicator

Exit Code:

• 0 – SYNCED (all tasks pass)
• non-zero – OUT_OF_SYNC (fail task reached)

Example output:

DRIFTED_FILES=/etc/rsyslog.conf,/etc/rsyslog.d/30-lab.conf
SYNC_STATUS=OUT_OF_SYNC

---

2. Shell Script: update-gitops-status.sh

Purpose: Orchestrates the complete status update workflow

Flow:

1. Step 1: Run drift-check.yml playbook

   • Captures stdout/stderr to temp file
   • Stores exit code

2. Step 2: Parse playbook output

   • Extract DRIFTED_FILES=... line
   • Parse comma-separated files
   • Convert full paths to relative paths
   • Determine sync status from exit code

3. Step 3: Build JSON payload

   • Repo name and server name
   • Sync status (SYNCED / OUT_OF_SYNC)
   • Drift count (number of changed files)
   • Array of changed files with names
   • ISO 8601 timestamp

4. Step 4: POST JSON to gitops-status-server

   • Endpoint: $GITOPS_STATUS_SERVER_URL/api/status
   • Method: POST
   • Content-Type: application/json
   • Check HTTP response code (200-299 = success)

Environment Variables:

GITOPS_STATUS_SERVER_URL=http://gitops-status-server.observability-stack.svc.cluster.local:80
REPO_NAME=rsyslog
SERVER_NAME=rsyslog-lab

JSON Payload Generated:

{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "SYNCED",
  "drift_count": 0,
  "files": [],
  "last_check": "2026-04-21T10:30:00Z"
}

Exit Codes:

• 0 – Success (JSON posted to gitops-status-server)
• 1 – Failure (playbook error, network error, etc.)

---

3. CI/CD Pipeline: .woodpecker.yml

Two integration points:

a) Post-Deployment (update-gitops-status step)

Runs after successful deployment on push to master:

update-gitops-status:
  depends_on: [deploy]
  when:
    branch: master
    event: push
  commands:
    - chmod +x update-gitops-status.sh
    - ./update-gitops-status.sh

Purpose: Verify deployment and update status immediately

b) Scheduled Drift Check (gitops_sync_check cron)

Runs on schedule (every 2 minutes by default):

gitops_sync_check:
  when:
    event: cron
  commands:
    - chmod +x update-gitops-status.sh
    - ./update-gitops-status.sh
    # Exit with appropriate code so cron shows as success/failed
    - ansible-playbook ... # re-run to determine exit code

Purpose: Continuous monitoring of server state

Woodpecker UI Setup:

1. Go to repository settings
2. Add cron job:
   • Name: gitops_sync_check
   • Branch: master
   • Schedule: */2 * * * * (every 2 minutes)

---

Data Flow

Input (from Ansible)

Drift-check playbook outputs structured markers:

...
DRIFTED_FILES=/etc/rsyslog.conf,/etc/rsyslog.d/30-lab.conf
SYNC_STATUS=OUT_OF_SYNC
...

Processing

Shell script parses output:

• Extract drifted files
• Convert paths: /etc/rsyslog.conf → rsyslog.conf
• Count changed files
• Get current timestamp

Output (to gitops-status-server)

{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "OUT_OF_SYNC",
  "drift_count": 2,
  "files": [
    { "name": "rsyslog.conf" },
    { "name": "rsyslog.d/30-lab.conf" }
  ],
  "last_check": "2026-04-21T10:30:00Z"
}

Final Display (in Grafana)

Grafana's Infinity datasource reads /status.json from gitops-status-server and displays:

• Sync Status: Visual indicator (green ✓ SYNCED / red ✗ OUT_OF_SYNC)
• Drift Count: Number of changed files
• Files: List of which files are different
• Last Check: Timestamp of last drift-check run

---

Workflow: Manual Server Changes Detection

Scenario: Someone manually edits /etc/rsyslog.conf directly on the server

Detection flow:

1. File is edited on server, Git repo remains unchanged
2. Cron timer triggers gitops_sync_check (every 2 minutes)
3. Woodpecker runs update-gitops-status.sh
4. Script executes drift-check.yml playbook
5. Ansible copy task in check mode detects difference
6. Playbook outputs: DRIFTED_FILES=/etc/rsyslog.conf
7. Script parses output and generates JSON
8. Script POSTs JSON to gitops-status-server
9. gitops-status-server updates its /status.json
10. Grafana Infinity datasource refreshes
11. Dashboard shows:
    • Status: OUT_OF_SYNC
    • Drift count: 1
    • Files: [rsyslog.conf]
    • Last check:

Time to detection: ≤ 2 minutes (next cron run)

---

Workflow: Deployment & Verification

Scenario: Code is pushed to master branch

Deployment flow:

1. Push to master triggers Woodpecker pipeline
2. Steps: syntax-check → validate → deploy → update-gitops-status
3. Deploy step runs ansible/playbooks/apply.yml
   • Copies Git files to server
   • Makes rsyslog.conf and rsyslog.d/* files match Git
4. Update-gitops-status step runs:
   • Executes drift-check.yml again
   • Should report no drift (files match Git)
   • Generates JSON with SYNC_STATUS=SYNCED
   • POSTs JSON to gitops-status-server
5. gitops-status-server updates /status.json to SYNCED
6. Grafana dashboard immediately shows SYNCED status

Expected result: After deployment, status should be SYNCED within minutes

---

Configuration

Environment Variables

Set in .woodpecker.yml:

environment:
  # URL of gitops-status-server ClusterIP service
  GITOPS_STATUS_SERVER_URL: http://gitops-status-server.observability-stack.svc.cluster.local:80
  
  # Repository identifier (for multi-repo dashboards)
  REPO_NAME: rsyslog
  
  # Server identifier (for multi-server dashboards)
  SERVER_NAME: rsyslog-lab
  
  # SSH private key (for Ansible to connect to server)
  SSH_PRIVATE_KEY: from_secret: SSH_PRIVATE_KEY
  
  # Ansible config location
  ANSIBLE_CONFIG: ansible.cfg

Customization

To support multiple servers:

gitops_sync_check:
  commands:
    - bash update-gitops-status.sh  # Uses env vars SERVER_NAME
    # Or:
    - SERVER_NAME=rsyslog-lab-1 bash update-gitops-status.sh
    - SERVER_NAME=rsyslog-lab-2 bash update-gitops-status.sh

---

Advantages Over Pushgateway

Aspect	Pushgateway	gitops-status-server
Infrastructure	Prometheus + Pushgateway	Single Kubernetes service
Data richness	Only 0/1 metric	Full JSON with file names, timestamps
Query language	PromQL (complex)	Simple JSON API (easy)
Grafana integration	Prometheus datasource	Infinity datasource (JSON)
Multi-repository	Complex labels	Built-in support
File-level details	Not available	Full list of changed files
Audit trail	Metrics only	JSON snapshot with metadata

---

Troubleshooting

JSON not being sent to gitops-status-server

Check:

1. Verify gitops-status-server is running:

   kubectl get pod -n observability-stack | grep gitops-status
   kubectl logs -n observability-stack <pod-name>
   

2. Test connectivity from Woodpecker:

   # In Woodpecker log, should see:
   # ==> Sending status to gitops-status-server...
   # ==> Response: HTTP 200
   

3. Check network connectivity:

   # From Woodpecker container, test the endpoint:
   curl http://gitops-status-server.observability-stack.svc.cluster.local:80/api/status
   

Drift not being detected

Check:

1. Review drift-check.yml output in Woodpecker logs:

   • Should see "✓ SYNCED" or "✗ OUT OF SYNC"
   • Should see "DRIFTED_FILES=" line

2. Manual test on server:

   ansible-playbook -i ansible/inventory/hosts.yml ansible/playbooks/drift-check.yml -v
   

3. Check SSH connectivity:

   # Verify SSH key is properly set in Woodpecker
   # Check server's rsyslog files are readable:
   ssh user@server ls -la /etc/rsyslog.conf /etc/rsyslog.d/
   

Cron job not running

Check:

1. In Woodpecker UI:

   • Go to repository settings
   • Click "Cron" or look for cron job list
   • Verify gitops_sync_check is listed with */2 * * * * schedule

2. Check cron execution history:

   • Woodpecker should show execution log
   • Look for "Step 1/4: Running drift-check playbook..."

3. Manual trigger:

   • In Woodpecker UI, try to manually trigger the cron job
   • Check logs for errors

Grafana not showing data

Check:

1. Verify gitops-status-server is serving JSON:

   curl http://gitops-status-server.observability-stack.svc.cluster.local:80/status.json
   

2. Check Grafana Infinity datasource configuration:

   • Datasource URL should point to gitops-status-server
   • Test button should show "Data source is working"

3. In dashboard:

   • Edit panel
   • Query should be: GET /status.json (or similar)
   • Click "Run query" to test
   • Should see JSON response with data

---

Security Considerations

• ✓ SSH credentials in Woodpecker secrets (not exposed)
• ✓ JSON contains only file names and metadata (not config contents)
• ✓ gitops-status-server only accepts POST from authorized CI/CD
• ✓ No actual rsyslog config files are exposed
• ✓ Network communication is internal (ClusterIP)

---

Monitoring

Key Metrics

• Cron execution: Every 2 minutes
• JSON update frequency: Every 2 minutes (or after deployment)
• Time to detection: ≤ 2 minutes for manual drift
• Time to verification: Immediate after deployment (post-deploy step)

Grafana Dashboard

• Status panel: Shows SYNCED / OUT_OF_SYNC
• Drift count: Number of changed files
• Changed files table: Lists affected files
• Last check: Timestamp of last check
• Repo/Server info: Identifies which repo/server

---

Examples

Example 1: Server is synced with Git

{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "SYNCED",
  "drift_count": 0,
  "files": [],
  "last_check": "2026-04-21T10:30:00Z"
}

Example 2: Manual edit on server

{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "OUT_OF_SYNC",
  "drift_count": 1,
  "files": [
    { "name": "rsyslog.conf" }
  ],
  "last_check": "2026-04-21T10:32:00Z"
}

Example 3: Multiple files changed

{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "OUT_OF_SYNC",
  "drift_count": 3,
  "files": [
    { "name": "rsyslog.conf" },
    { "name": "rsyslog.d/30-lab.conf" },
    { "name": "rsyslog.d/31-remote.conf" }
  ],
  "last_check": "2026-04-21T10:34:00Z"
}

---

Summary

This integration provides:

• ✓ Real-time drift detection via cron (every 2 minutes)
• ✓ Post-deployment verification
• ✓ File-level granularity (which files changed)
• ✓ Integration with gitops-status-server (centralized service)
• ✓ Grafana Infinity datasource support
• ✓ Clean JSON-based architecture
• ✓ No Pushgateway dependency for this use case

The rsyslog repository is now responsible for producing clean, structured JSON snapshots of its GitOps status, which are consumed by gitops-status-server and displayed in Grafana.