229 lines
10 KiB
Markdown
229 lines
10 KiB
Markdown
# rsyslog GitOps
|
|
|
|
Manage rsyslog configuration on Linux servers using Git as the single source of truth.
|
|
If it's not in Git, it doesn't belong on the server.
|
|
|
|
---
|
|
|
|
## How it works in one sentence
|
|
|
|
Every change goes through Git. The pipeline makes sure the server always matches what's in Git — and if someone changes the server directly, the system detects it automatically.
|
|
|
|
---
|
|
|
|
## The three pipelines
|
|
|
|
### 1. Pull Request — "Is this config safe?"
|
|
|
|
Triggered when you open or update a pull request.
|
|
Does **not** touch the live server beyond a basic reachability check.
|
|
Does **not** compare the PR content to the server (they're expected to differ before merge).
|
|
|
|
```
|
|
Open PR
|
|
│
|
|
├─► syntax-check Check the YAML/Ansible syntax is valid
|
|
│
|
|
└─► validate Connect to the server and verify rsyslog is running
|
|
and the current config is loadable
|
|
```
|
|
|
|
**Pass** = safe to review and merge.
|
|
**Fail** = syntax error or server is unreachable / config is broken.
|
|
|
|
---
|
|
|
|
### 2. Push to master — "Deploy and verify"
|
|
|
|
Triggered when a PR is merged into master.
|
|
|
|
```
|
|
Merge to master
|
|
│
|
|
├─► syntax-check Same lint check as PR
|
|
│
|
|
├─► validate Same server check as PR
|
|
│
|
|
├─► deploy Copy the new config files from Git to the server
|
|
│ and restart rsyslog
|
|
│
|
|
└─► update-gitops-status Run a diff between Git and the live server
|
|
│
|
|
├─ Matches? → send JSON (SYNCED, drift_count: 0)
|
|
└─ Differs? → send JSON (OUT_OF_SYNC, drift_count: N, files: [...])
|
|
│
|
|
└─ Update gitops-status-server for Grafana visualization
|
|
```
|
|
|
|
**Pass** = new config is live and the server matches Git.
|
|
The sync status JSON is always sent to gitops-status-server regardless of outcome.
|
|
|
|
---
|
|
|
|
### 3. Cron — "Is the server still synced?"
|
|
|
|
Runs automatically every 2 minutes, **even with no new push**.
|
|
This is the ArgoCD-style continuous check.
|
|
It only reads — never deploys anything.
|
|
|
|
```
|
|
Every 2 minutes (cron)
|
|
│
|
|
└─► gitops_sync_check SSH to the server, compare every managed config
|
|
file against the latest Git commit
|
|
│
|
|
├─ Matches? → send JSON (SYNCED, drift_count: 0, files: [])
|
|
└─ Differs? → send JSON (OUT_OF_SYNC, drift_count: N, files: [...])
|
|
│
|
|
└─ Update gitops-status-server for Grafana visualization
|
|
```
|
|
|
|
**Why this matters:** if someone edits `/etc/rsyslog.conf` directly on the server
|
|
(bypassing Git), the next cron run catches it within 2 minutes and marks OUT_OF_SYNC
|
|
with detailed information about which specific files have drifted.
|
|
|
|
---
|
|
|
|
## Full flow diagramgitops-status-server
|
|
│ │ │ │
|
|
│── open PR ───────────────►│ │ │
|
|
│ │── syntax-check │ │
|
|
│ │── validate ─────────────►│ │
|
|
│◄── PR ok / failed ────────│ │ │
|
|
│ │ │ │
|
|
│── merge to master ───────►│ │ │
|
|
│ │── syntax-check │ │
|
|
│ │── validate ─────────────►│ │
|
|
│ │── deploy ───────────────►│ write config │
|
|
│ │ │ restart rsyslog │
|
|
│ │── drift-check ──────────►│ compare files │
|
|
│ │ │◄────────────────────│
|
|
│ │── JSON status ───────────────────────────────►│
|
|
│ │ │ │
|
|
│ │ [every 2 min, no push] │ │
|
|
│ │── drift-check ──────────►│ compare files │
|
|
│ │── JSON status ───────────────────────────────►│
|
|
│ │ │ │
|
|
|
|
|
|
Someone edits the server directly (bad):
|
|
|
|
rogue admin Woodpecker CI Linux Server gitops-status-server
|
|
│ │ │ │
|
|
│── ssh rsyslog-lab │ │ │
|
|
│── vim /etc/rsyslog.conf ──────────────────────────► │ file changed │
|
|
│ │ │ │
|
|
│ [2 min later, cron runs] │ │
|
|
│ │── drift-check ──────────►│ diff detected │
|
|
│ │── JSON status (OUT_OF_SYNC)─────────────────►│
|
|
│ │ drift_count: 1 │ Grafana shows
|
|
│ │ files: [rsyslog.conf] OUT_OF_SYNC
|
|
│ │── drift-check ──────────►│ diff detected │
|
|
│ │── metric 0 (OUT_OF_SYNC)────────────────────►│
|
|
│ │ │ alert fires
|
|
```
|
|
|
|
---
|
|
GitOps status JSON format
|
|
|
|
Instead of simple numeric metrics, this repo now sends rich JSON status data to gitops-status-server:
|
|
|
|
```json
|
|
{
|
|
"repo": "rsyslog",
|
|
"server": "rsyslog-lab",
|
|
"sync_status": "SYNCED",
|
|
"drift_count": 0,
|
|
"files": [],
|
|
"last_check": "2026-04-21T10:30:00Z"
|
|
}
|
|
```
|
|
|
|
When drift is detected:
|
|
|
|
```json
|
|
{
|
|
"repo": "rsyslog",
|
|
"server": "rsyslog-lab",
|
|
"sync_status": "OUT_OF_SYNC",
|
|
"drift_count": 2,
|
|
"files": [
|
|
{ "name": "rsyslog.conf" },
|
|
{ "name": "rsyslog.d/30-lab.conf" }
|
|
],
|
|
"last_check": "2026-04-21T10:32:15Z"
|
|
}
|
|
```
|
|
|
|
update-gitops-status.sh Script to generate and send JSON status to gitops-status-server
|
|
This JSON is sent to `gitops-status-server` at:
|
|
- `http://gitops-status-server.observability-stack.svc.cluster.local:80/api/status`
|
|
|
|
The gitops-status-server app serves this data via `/status.json` for Grafana Infinity datasource,
|
|
providing rich visualization with file-level drift details instead of just a numeric metric
|
|
Alert on `gitops_sync_status == 0` in Grafana/Alertmanager.
|
|
|
|
---
|
|
|
|
## What drift-check actually compares
|
|
|
|
The drift-check playbook compares files **from the Woodpecker CI container** (always the latest Git commit) against the live server. It checks:
|
|
|
|
1. `/etc/rsyslog.conf` — must match `files/rsyslog.conf` in Git
|
|
2. `/etc/rsyslog.d/30-lab.conf` — must match `files/rsyslog.d/30-lab.conf` in Git
|
|
3. Any file managed by Git must not be missing from the server
|
|
|
|
Files on the server that are **not** in Git (e.g. `50-default.conf`, `20-ufw.conf`) are ignored — they are owned by the OS and are not our concern.
|
|
|
|
---
|
|
|
|
## Repository structure
|
|
|
|
```
|
|
.woodpecker.yml CI pipeline definition
|
|
ansible/
|
|
inventory/
|
|
hosts.yml Server list
|
|
group_vars/all.yml Variables (paths, user, etc.)
|
|
playbooks/
|
|
validate.yml Check rsyslog is running and config loads
|
|
apply.yml Deploy config files from Git to server
|
|
drift-check.yml Compare Git files to live server (read-only)
|
|
files/
|
|
rsyslog.conf Main rsyslog config (source of truth)
|
|
rsyslog.d/
|
|
30-lab.conf Drop-in config for lab logging
|
|
```
|
|
|
|
---
|
|
|
|
## Woodpecker cron setup
|
|
|
|
Go to **Repository Settings → Crons → Add cron**:
|
|
|
|
| Field | Value |
|
|
|----------|---------------------|
|
|
| Name | `gitops_sync_check` |
|
|
| Branch | `master` |
|
|
| Schedule | `*/2 * * * *` |
|
|
|
|
---
|
|
|
|
## Required secrets
|
|
|
|
Go to **Repository Settings → Secrets**:
|
|
|
|
| Name | Description |
|
|
|----------------------------|-------------------------------------------------------|
|
|
| `SSH_PRIVATE_KEY` | Private key to SSH into the server |
|
|
|
|
## Optional environment variables
|
|
|
|
These can be overridden in the Woodpecker pipeline or `.woodpecker.yml`:
|
|
|
|
| Variable | Default | Description |
|
|
|------------------------------|--------------------------------------------------------------------------|---------------------------------------|
|
|
| `GITOPS_STATUS_SERVER_URL` | `http://gitops-status-server.observability-stack.svc.cluster.local:80` | URL of gitops-status-server API |
|
|
| `REPO_NAME` | `rsyslog` | Repository name for JSON status |
|
|
| `SERVER_NAME` | `rsyslog-lab` | Server name for JSON status |
|