# rsyslog GitOps Manage rsyslog configuration on Linux servers using Git as the single source of truth. If it's not in Git, it doesn't belong on the server. --- ## How it works in one sentence Every change goes through Git. The pipeline makes sure the server always matches what's in Git — and if someone changes the server directly, the system detects it automatically. --- ## The three pipelines ### 1. Pull Request — "Is this config safe?" Triggered when you open or update a pull request. Does **not** touch the live server beyond a basic reachability check. Does **not** compare the PR content to the server (they're expected to differ before merge). ``` Open PR │ ├─► syntax-check Check the YAML/Ansible syntax is valid │ └─► validate Connect to the server and verify rsyslog is running and the current config is loadable ``` **Pass** = safe to review and merge. **Fail** = syntax error or server is unreachable / config is broken. --- ### 2. Push to master — "Deploy and verify" Triggered when a PR is merged into master. ``` Merge to master │ ├─► syntax-check Same lint check as PR │ ├─► validate Same server check as PR │ ├─► deploy Copy the new config files from Git to the server │ and restart rsyslog │ └─► update-gitops-status Run a diff between Git and the live server │ ├─ Matches? → send JSON (SYNCED, drift_count: 0) └─ Differs? → send JSON (OUT_OF_SYNC, drift_count: N, files: [...]) │ └─ Update gitops-status-server for Grafana visualization ``` **Pass** = new config is live and the server matches Git. The sync status JSON is always sent to gitops-status-server regardless of outcome. --- ### 3. Cron — "Is the server still synced?" Runs automatically every 2 minutes, **even with no new push**. This is the ArgoCD-style continuous check. It only reads — never deploys anything. ``` Every 2 minutes (cron) │ └─► gitops_sync_check SSH to the server, compare every managed config file against the latest Git commit │ ├─ Matches? → send JSON (SYNCED, drift_count: 0, files: []) └─ Differs? → send JSON (OUT_OF_SYNC, drift_count: N, files: [...]) │ └─ Update gitops-status-server for Grafana visualization ``` **Why this matters:** if someone edits `/etc/rsyslog.conf` directly on the server (bypassing Git), the next cron run catches it within 2 minutes and marks OUT_OF_SYNC with detailed information about which specific files have drifted. --- ## Full flow diagramgitops-status-server │ │ │ │ │── open PR ───────────────►│ │ │ │ │── syntax-check │ │ │ │── validate ─────────────►│ │ │◄── PR ok / failed ────────│ │ │ │ │ │ │ │── merge to master ───────►│ │ │ │ │── syntax-check │ │ │ │── validate ─────────────►│ │ │ │── deploy ───────────────►│ write config │ │ │ │ restart rsyslog │ │ │── drift-check ──────────►│ compare files │ │ │ │◄────────────────────│ │ │── JSON status ───────────────────────────────►│ │ │ │ │ │ │ [every 2 min, no push] │ │ │ │── drift-check ──────────►│ compare files │ │ │── JSON status ───────────────────────────────►│ │ │ │ │ Someone edits the server directly (bad): rogue admin Woodpecker CI Linux Server gitops-status-server │ │ │ │ │── ssh rsyslog-lab │ │ │ │── vim /etc/rsyslog.conf ──────────────────────────► │ file changed │ │ │ │ │ │ [2 min later, cron runs] │ │ │ │── drift-check ──────────►│ diff detected │ │ │── JSON status (OUT_OF_SYNC)─────────────────►│ │ │ drift_count: 1 │ Grafana shows │ │ files: [rsyslog.conf] OUT_OF_SYNC │ │── drift-check ──────────►│ diff detected │ │ │── metric 0 (OUT_OF_SYNC)────────────────────►│ │ │ │ alert fires ``` --- GitOps status JSON format Instead of simple numeric metrics, this repo now sends rich JSON status data to gitops-status-server: ```json { "repo": "rsyslog", "server": "rsyslog-lab", "sync_status": "SYNCED", "drift_count": 0, "files": [], "last_check": "2026-04-21T10:30:00Z" } ``` When drift is detected: ```json { "repo": "rsyslog", "server": "rsyslog-lab", "sync_status": "OUT_OF_SYNC", "drift_count": 2, "files": [ { "name": "rsyslog.conf" }, { "name": "rsyslog.d/30-lab.conf" } ], "last_check": "2026-04-21T10:32:15Z" } ``` update-gitops-status.sh Script to generate and send JSON status to gitops-status-server This JSON is sent to `gitops-status-server` at: - `http://gitops-status-server.observability-stack.svc.cluster.local:80/api/status` The gitops-status-server app serves this data via `/status.json` for Grafana Infinity datasource, providing rich visualization with file-level drift details instead of just a numeric metric Alert on `gitops_sync_status == 0` in Grafana/Alertmanager. --- ## What drift-check actually compares The drift-check playbook compares files **from the Woodpecker CI container** (always the latest Git commit) against the live server. It checks: 1. `/etc/rsyslog.conf` — must match `files/rsyslog.conf` in Git 2. `/etc/rsyslog.d/30-lab.conf` — must match `files/rsyslog.d/30-lab.conf` in Git 3. Any file managed by Git must not be missing from the server Files on the server that are **not** in Git (e.g. `50-default.conf`, `20-ufw.conf`) are ignored — they are owned by the OS and are not our concern. --- ## Repository structure ``` .woodpecker.yml CI pipeline definition ansible/ inventory/ hosts.yml Server list group_vars/all.yml Variables (paths, user, etc.) playbooks/ validate.yml Check rsyslog is running and config loads apply.yml Deploy config files from Git to server drift-check.yml Compare Git files to live server (read-only) files/ rsyslog.conf Main rsyslog config (source of truth) rsyslog.d/ 30-lab.conf Drop-in config for lab logging ``` --- ## Woodpecker cron setup Go to **Repository Settings → Crons → Add cron**: | Field | Value | |----------|---------------------| | Name | `gitops_sync_check` | | Branch | `master` | | Schedule | `*/2 * * * *` | --- ## Required secrets Go to **Repository Settings → Secrets**: | Name | Description | |----------------------------|-------------------------------------------------------| | `SSH_PRIVATE_KEY` | Private key to SSH into the server | ## Optional environment variables These can be overridden in the Woodpecker pipeline or `.woodpecker.yml`: | Variable | Default | Description | |------------------------------|--------------------------------------------------------------------------|---------------------------------------| | `GITOPS_STATUS_SERVER_URL` | `http://gitops-status-server.observability-stack.svc.cluster.local:80` | URL of gitops-status-server API | | `REPO_NAME` | `rsyslog` | Repository name for JSON status | | `SERVER_NAME` | `rsyslog-lab` | Server name for JSON status |