diff --git a/README.md b/README.md new file mode 100644 index 0000000..a84704f --- /dev/null +++ b/README.md @@ -0,0 +1,187 @@ +# rsyslog GitOps + +Manage rsyslog configuration on Linux servers using Git as the single source of truth. +If it's not in Git, it doesn't belong on the server. + +--- + +## How it works in one sentence + +Every change goes through Git. The pipeline makes sure the server always matches what's in Git — and if someone changes the server directly, the system detects it automatically. + +--- + +## The three pipelines + +### 1. Pull Request — "Is this config safe?" + +Triggered when you open or update a pull request. +Does **not** touch the live server beyond a basic reachability check. +Does **not** compare the PR content to the server (they're expected to differ before merge). + +``` +Open PR + │ + ├─► syntax-check Check the YAML/Ansible syntax is valid + │ + └─► validate Connect to the server and verify rsyslog is running + and the current config is loadable +``` + +**Pass** = safe to review and merge. +**Fail** = syntax error or server is unreachable / config is broken. + +--- + +### 2. Push to master — "Deploy and verify" + +Triggered when a PR is merged into master. + +``` +Merge to master + │ + ├─► syntax-check Same lint check as PR + │ + ├─► validate Same server check as PR + │ + ├─► deploy Copy the new config files from Git to the server + │ and restart rsyslog + │ + └─► update-sync-metric Run a diff between Git and the live server + │ + ├─ Matches? → push metric 1 (SYNCED) + └─ Differs? → push metric 0 (OUT_OF_SYNC) +``` + +**Pass** = new config is live and the server matches Git. +The sync result is always sent to Prometheus regardless of outcome. + +--- + +### 3. Cron — "Is the server still synced?" + +Runs automatically every 2 minutes, **even with no new push**. +This is the ArgoCD-style continuous check. +It only reads — never deploys anything. + +``` +Every 2 minutes (cron) + │ + └─► gitops_sync_check SSH to the server, compare every managed config + file against the latest Git commit + │ + ├─ Matches? → push metric 1 (SYNCED) + └─ Differs? → push metric 0 (OUT_OF_SYNC) +``` + +**Why this matters:** if someone edits `/etc/rsyslog.conf` directly on the server +(bypassing Git), the next cron run catches it within 2 minutes and marks OUT_OF_SYNC. + +--- + +## Full flow diagram + +``` +Developer Woodpecker CI Linux Server Prometheus + │ │ │ │ + │── open PR ───────────────►│ │ │ + │ │── syntax-check │ │ + │ │── validate ─────────────►│ │ + │◄── PR ok / failed ────────│ │ │ + │ │ │ │ + │── merge to master ───────►│ │ │ + │ │── syntax-check │ │ + │ │── validate ─────────────►│ │ + │ │── deploy ───────────────►│ write config │ + │ │ │ restart rsyslog │ + │ │── drift-check ──────────►│ compare files │ + │ │ │◄────────────────────│ + │ │── metric (1 or 0) ───────────────────────────►│ + │ │ │ │ + │ │ [every 2 min, no push] │ │ + │ │── drift-check ──────────►│ compare files │ + │ │── metric (1 or 0) ───────────────────────────►│ + │ │ │ │ + + +Someone edits the server directly (bad): + + rogue admin Woodpecker CI Linux Server Prometheus + │ │ │ │ + │── ssh rsyslog-lab │ │ │ + │── vim /etc/rsyslog.conf ──────────────────────────► │ file changed │ + │ │ │ │ + │ [2 min later, cron runs] │ │ + │ │── drift-check ──────────►│ diff detected │ + │ │── metric 0 (OUT_OF_SYNC)────────────────────►│ + │ │ │ alert fires +``` + +--- + +## What is the sync metric? + +``` +gitops_sync_status{repo="rsyslog", server="rsyslog-lab"} +``` + +| Value | Meaning | +|-------|---------| +| `1` | Server config matches Git (SYNCED) | +| `0` | Server config differs from Git (OUT_OF_SYNC) | + +Alert on `gitops_sync_status == 0` in Grafana/Alertmanager. + +--- + +## What drift-check actually compares + +The drift-check playbook compares files **from the Woodpecker CI container** (always the latest Git commit) against the live server. It checks: + +1. `/etc/rsyslog.conf` — must match `files/rsyslog.conf` in Git +2. `/etc/rsyslog.d/30-lab.conf` — must match `files/rsyslog.d/30-lab.conf` in Git +3. Any file managed by Git must not be missing from the server + +Files on the server that are **not** in Git (e.g. `50-default.conf`, `20-ufw.conf`) are ignored — they are owned by the OS and are not our concern. + +--- + +## Repository structure + +``` +.woodpecker.yml CI pipeline definition +ansible/ + inventory/ + hosts.yml Server list + group_vars/all.yml Variables (paths, user, etc.) + playbooks/ + validate.yml Check rsyslog is running and config loads + apply.yml Deploy config files from Git to server + drift-check.yml Compare Git files to live server (read-only) +files/ + rsyslog.conf Main rsyslog config (source of truth) + rsyslog.d/ + 30-lab.conf Drop-in config for lab logging +``` + +--- + +## Woodpecker cron setup + +Go to **Repository Settings → Crons → Add cron**: + +| Field | Value | +|----------|---------------------| +| Name | `gitops_sync_check` | +| Branch | `master` | +| Schedule | `*/2 * * * *` | + +--- + +## Required secrets + +Go to **Repository Settings → Secrets**: + +| Name | Description | +|-------------------|------------------------------------| +| `SSH_PRIVATE_KEY` | Private key to SSH into the server |