rsyslog GitOps

Manage rsyslog configuration on Linux servers using Git as the single source of truth.
If it's not in Git, it doesn't belong on the server.

---

How it works in one sentence

Every change goes through Git. The pipeline makes sure the server always matches what's in Git — and if someone changes the server directly, the system detects it automatically.

---

The three pipelines

1. Pull Request — "Is this config safe?"

Triggered when you open or update a pull request.
Does not touch the live server beyond a basic reachability check.
Does not compare the PR content to the server (they're expected to differ before merge).

Open PR
   │
   ├─► syntax-check   Check the YAML/Ansible syntax is valid
   │
   └─► validate       Connect to the server and verify rsyslog is running
                      and the current config is loadable

Pass = safe to review and merge.
Fail = syntax error or server is unreachable / config is broken.

---

2. Push to master — "Deploy and verify"

Triggered when a PR is merged into master.

Merge to master
   │
   ├─► syntax-check           Same lint check as PR
   │
   ├─► validate               Same server check as PR
   │
   ├─► deploy                 Copy the new config files from Git to the server
   │                          and restart rsyslog
   │
   └─► update-gitops-status   Run a diff between Git and the live server
                              │
                              ├─ Matches?  → send JSON (SYNCED, drift_count: 0)
                              └─ Differs?  → send JSON (OUT_OF_SYNC, drift_count: N, files: [...])
                              │
                              └─ Update gitops-status-server for Grafana visualization

Pass = new config is live and the server matches Git.
The sync status JSON is always sent to gitops-status-server regardless of outcome.

---

3. Cron — "Is the server still synced?"

Runs automatically every 2 minutes, even with no new push.
This is the ArgoCD-style continuous check.
It only reads — never deploys anything.

Every 2 minutes (cron)
   │
   └─► gitops_sync_check   SSH to the server, compare every managed config
                           file against the latest Git commit
                           │
                           ├─ Matches?  → send JSON (SYNCED, drift_count: 0, files: [])
                           └─ Differs?  → send JSON (OUT_OF_SYNC, drift_count: N, files: [...])
                           │
                           └─ Update gitops-status-server for Grafana visualization

Why this matters: if someone edits /etc/rsyslog.conf directly on the server
(bypassing Git), the next cron run catches it within 2 minutes and marks OUT_OF_SYNC
with detailed information about which specific files have drifted.

---

Full flow diagramgitops-status-server

│                           │                          │                     │
│── open PR ───────────────►│                          │                     │
│                           │── syntax-check           │                     │
│                           │── validate ─────────────►│                     │
│◄── PR ok / failed ────────│                          │                     │
│                           │                          │                     │
│── merge to master ───────►│                          │                     │
│                           │── syntax-check           │                     │
│                           │── validate ─────────────►│                     │
│                           │── deploy ───────────────►│ write config        │
│                           │                          │ restart rsyslog     │
│                           │── drift-check ──────────►│ compare files       │
│                           │                          │◄────────────────────│
│                           │── JSON status ───────────────────────────────►│
│                           │                          │                     │
│                           │  [every 2 min, no push]  │                     │
│                           │── drift-check ──────────►│ compare files       │
│                           │── JSON status ───────────────────────────────►│
│                           │                          │                     │

Someone edits the server directly (bad):

rogue admin                 Woodpecker CI              Linux Server       gitops-status-server
     │                           │                          │                     │
     │── ssh rsyslog-lab         │                          │                     │
     │── vim /etc/rsyslog.conf ──────────────────────────► │ file changed        │
     │                           │                          │                     │
     │                     [2 min later, cron runs]        │                     │
     │                           │── drift-check ──────────►│ diff detected       │
     │                           │── JSON status (OUT_OF_SYNC)─────────────────►│
     │                           │        drift_count: 1    │              Grafana shows
     │                           │        files: [rsyslog.conf]          OUT_OF_SYNC
     │                           │── drift-check ──────────►│ diff detected       │
     │                           │── metric 0 (OUT_OF_SYNC)────────────────────►│
     │                           │                          │              alert fires

---
GitOps status JSON format

Instead of simple numeric metrics, this repo now sends rich JSON status data to gitops-status-server:

```json
{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "SYNCED",
  "drift_count": 0,
  "files": [],
  "last_check": "2026-04-21T10:30:00Z"
}

When drift is detected:

{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "OUT_OF_SYNC",
  "drift_count": 2,
  "files": [
    { "name": "rsyslog.conf" },
    { "name": "rsyslog.d/30-lab.conf" }
  ],
  "last_check": "2026-04-21T10:32:15Z"
}

update-gitops-status.sh Script to generate and send JSON status to gitops-status-server This JSON is sent to gitops-status-server at:

• http://gitops-status-server.observability-stack.svc.cluster.local:80/api/status

The gitops-status-server app serves this data via /status.json for Grafana Infinity datasource,
providing rich visualization with file-level drift details instead of just a numeric metric Alert on gitops_sync_status == 0 in Grafana/Alertmanager.

---

What drift-check actually compares

The drift-check playbook compares files from the Woodpecker CI container (always the latest Git commit) against the live server. It checks:

1. /etc/rsyslog.conf — must match files/rsyslog.conf in Git
2. /etc/rsyslog.d/30-lab.conf — must match files/rsyslog.d/30-lab.conf in Git
3. Any file managed by Git must not be missing from the server

Files on the server that are not in Git (e.g. 50-default.conf, 20-ufw.conf) are ignored — they are owned by the OS and are not our concern.

---

Repository structure

.woodpecker.yml              CI pipeline definition
ansible/
  inventory/
    hosts.yml                Server list
    group_vars/all.yml       Variables (paths, user, etc.)
  playbooks/
    validate.yml             Check rsyslog is running and config loads
    apply.yml                Deploy config files from Git to server
    drift-check.yml          Compare Git files to live server (read-only)
files/
  rsyslog.conf               Main rsyslog config (source of truth)
  rsyslog.d/
    30-lab.conf              Drop-in config for lab logging

---

Woodpecker cron setup

Go to Repository Settings → Crons → Add cron:

Field	Value
Name	gitops_sync_check
Branch	master
Schedule	*/2 * * * *

---

Required secrets

Go to Repository Settings → Secrets:

Name	Description
SSH_PRIVATE_KEY	Private key to SSH into the server

Optional environment variables

These can be overridden in the Woodpecker pipeline or .woodpecker.yml:

Variable	Default	Description
GITOPS_STATUS_SERVER_URL	http://gitops-status-server.observability-stack.svc.cluster.local:80	URL of gitops-status-server API
REPO_NAME	rsyslog	Repository name for JSON status
SERVER_NAME	rsyslog-lab	Server name for JSON status