188 lines
7.6 KiB
Markdown
188 lines
7.6 KiB
Markdown
# rsyslog GitOps
|
|
|
|
Manage rsyslog configuration on Linux servers using Git as the single source of truth.
|
|
If it's not in Git, it doesn't belong on the server.
|
|
|
|
---
|
|
|
|
## How it works in one sentence
|
|
|
|
Every change goes through Git. The pipeline makes sure the server always matches what's in Git — and if someone changes the server directly, the system detects it automatically.
|
|
|
|
---
|
|
|
|
## The three pipelines
|
|
|
|
### 1. Pull Request — "Is this config safe?"
|
|
|
|
Triggered when you open or update a pull request.
|
|
Does **not** touch the live server beyond a basic reachability check.
|
|
Does **not** compare the PR content to the server (they're expected to differ before merge).
|
|
|
|
```
|
|
Open PR
|
|
│
|
|
├─► syntax-check Check the YAML/Ansible syntax is valid
|
|
│
|
|
└─► validate Connect to the server and verify rsyslog is running
|
|
and the current config is loadable
|
|
```
|
|
|
|
**Pass** = safe to review and merge.
|
|
**Fail** = syntax error or server is unreachable / config is broken.
|
|
|
|
---
|
|
|
|
### 2. Push to master — "Deploy and verify"
|
|
|
|
Triggered when a PR is merged into master.
|
|
|
|
```
|
|
Merge to master
|
|
│
|
|
├─► syntax-check Same lint check as PR
|
|
│
|
|
├─► validate Same server check as PR
|
|
│
|
|
├─► deploy Copy the new config files from Git to the server
|
|
│ and restart rsyslog
|
|
│
|
|
└─► update-sync-metric Run a diff between Git and the live server
|
|
│
|
|
├─ Matches? → push metric 1 (SYNCED)
|
|
└─ Differs? → push metric 0 (OUT_OF_SYNC)
|
|
```
|
|
|
|
**Pass** = new config is live and the server matches Git.
|
|
The sync result is always sent to Prometheus regardless of outcome.
|
|
|
|
---
|
|
|
|
### 3. Cron — "Is the server still synced?"
|
|
|
|
Runs automatically every 2 minutes, **even with no new push**.
|
|
This is the ArgoCD-style continuous check.
|
|
It only reads — never deploys anything.
|
|
|
|
```
|
|
Every 2 minutes (cron)
|
|
│
|
|
└─► gitops_sync_check SSH to the server, compare every managed config
|
|
file against the latest Git commit
|
|
│
|
|
├─ Matches? → push metric 1 (SYNCED)
|
|
└─ Differs? → push metric 0 (OUT_OF_SYNC)
|
|
```
|
|
|
|
**Why this matters:** if someone edits `/etc/rsyslog.conf` directly on the server
|
|
(bypassing Git), the next cron run catches it within 2 minutes and marks OUT_OF_SYNC.
|
|
|
|
---
|
|
|
|
## Full flow diagram
|
|
|
|
```
|
|
Developer Woodpecker CI Linux Server Prometheus
|
|
│ │ │ │
|
|
│── open PR ───────────────►│ │ │
|
|
│ │── syntax-check │ │
|
|
│ │── validate ─────────────►│ │
|
|
│◄── PR ok / failed ────────│ │ │
|
|
│ │ │ │
|
|
│── merge to master ───────►│ │ │
|
|
│ │── syntax-check │ │
|
|
│ │── validate ─────────────►│ │
|
|
│ │── deploy ───────────────►│ write config │
|
|
│ │ │ restart rsyslog │
|
|
│ │── drift-check ──────────►│ compare files │
|
|
│ │ │◄────────────────────│
|
|
│ │── metric (1 or 0) ───────────────────────────►│
|
|
│ │ │ │
|
|
│ │ [every 2 min, no push] │ │
|
|
│ │── drift-check ──────────►│ compare files │
|
|
│ │── metric (1 or 0) ───────────────────────────►│
|
|
│ │ │ │
|
|
|
|
|
|
Someone edits the server directly (bad):
|
|
|
|
rogue admin Woodpecker CI Linux Server Prometheus
|
|
│ │ │ │
|
|
│── ssh rsyslog-lab │ │ │
|
|
│── vim /etc/rsyslog.conf ──────────────────────────► │ file changed │
|
|
│ │ │ │
|
|
│ [2 min later, cron runs] │ │
|
|
│ │── drift-check ──────────►│ diff detected │
|
|
│ │── metric 0 (OUT_OF_SYNC)────────────────────►│
|
|
│ │ │ alert fires
|
|
```
|
|
|
|
---
|
|
|
|
## What is the sync metric?
|
|
|
|
```
|
|
gitops_sync_status{repo="rsyslog", server="rsyslog-lab"}
|
|
```
|
|
|
|
| Value | Meaning |
|
|
|-------|---------|
|
|
| `1` | Server config matches Git (SYNCED) |
|
|
| `0` | Server config differs from Git (OUT_OF_SYNC) |
|
|
|
|
Alert on `gitops_sync_status == 0` in Grafana/Alertmanager.
|
|
|
|
---
|
|
|
|
## What drift-check actually compares
|
|
|
|
The drift-check playbook compares files **from the Woodpecker CI container** (always the latest Git commit) against the live server. It checks:
|
|
|
|
1. `/etc/rsyslog.conf` — must match `files/rsyslog.conf` in Git
|
|
2. `/etc/rsyslog.d/30-lab.conf` — must match `files/rsyslog.d/30-lab.conf` in Git
|
|
3. Any file managed by Git must not be missing from the server
|
|
|
|
Files on the server that are **not** in Git (e.g. `50-default.conf`, `20-ufw.conf`) are ignored — they are owned by the OS and are not our concern.
|
|
|
|
---
|
|
|
|
## Repository structure
|
|
|
|
```
|
|
.woodpecker.yml CI pipeline definition
|
|
ansible/
|
|
inventory/
|
|
hosts.yml Server list
|
|
group_vars/all.yml Variables (paths, user, etc.)
|
|
playbooks/
|
|
validate.yml Check rsyslog is running and config loads
|
|
apply.yml Deploy config files from Git to server
|
|
drift-check.yml Compare Git files to live server (read-only)
|
|
files/
|
|
rsyslog.conf Main rsyslog config (source of truth)
|
|
rsyslog.d/
|
|
30-lab.conf Drop-in config for lab logging
|
|
```
|
|
|
|
---
|
|
|
|
## Woodpecker cron setup
|
|
|
|
Go to **Repository Settings → Crons → Add cron**:
|
|
|
|
| Field | Value |
|
|
|----------|---------------------|
|
|
| Name | `gitops_sync_check` |
|
|
| Branch | `master` |
|
|
| Schedule | `*/2 * * * *` |
|
|
|
|
---
|
|
|
|
## Required secrets
|
|
|
|
Go to **Repository Settings → Secrets**:
|
|
|
|
| Name | Description |
|
|
|-------------------|------------------------------------|
|
|
| `SSH_PRIVATE_KEY` | Private key to SSH into the server |
|