# rsyslog GitOps

Manage rsyslog configuration on Linux servers using Git as the single source of truth.  
If it's not in Git, it doesn't belong on the server.

---

## How it works in one sentence

Every change goes through Git. The pipeline makes sure the server always matches what's in Git — and if someone changes the server directly, the system detects it automatically.

---

## The three pipelines

### 1. Pull Request — "Is this config safe?"

Triggered when you open or update a pull request.  
Does **not** touch the live server beyond a basic reachability check.  
Does **not** compare the PR content to the server (they're expected to differ before merge).

```
Open PR
   │
   ├─► syntax-check   Check the YAML/Ansible syntax is valid
   │
   └─► validate       Connect to the server and verify rsyslog is running
                      and the current config is loadable
```

**Pass** = safe to review and merge.  
**Fail** = syntax error or server is unreachable / config is broken.

---

### 2. Push to master — "Deploy and verify"

Triggered when a PR is merged into master.

```
Merge to master
   │
   ├─► syntax-check      Same lint check as PR
   │
   ├─► validate          Same server check as PR
   │
   ├─► deploy            Copy the new config files from Git to the server
   │                     and restart rsyslog
   │
   └─► update-sync-metric   Run a diff between Git and the live server
                             │
                             ├─ Matches?  → push metric 1 (SYNCED)
                             └─ Differs?  → push metric 0 (OUT_OF_SYNC)
```

**Pass** = new config is live and the server matches Git.  
The sync result is always sent to Prometheus regardless of outcome.

---

### 3. Cron — "Is the server still synced?"

Runs automatically every 2 minutes, **even with no new push**.  
This is the ArgoCD-style continuous check.  
It only reads — never deploys anything.

```
Every 2 minutes (cron)
   │
   └─► gitops_sync_check   SSH to the server, compare every managed config
                           file against the latest Git commit
                           │
                           ├─ Matches?  → push metric 1 (SYNCED)
                           └─ Differs?  → push metric 0 (OUT_OF_SYNC)
```

**Why this matters:** if someone edits `/etc/rsyslog.conf` directly on the server  
(bypassing Git), the next cron run catches it within 2 minutes and marks OUT_OF_SYNC.

---

## Full flow diagram

```
Developer                  Woodpecker CI              Linux Server          Prometheus
    │                           │                          │                     │
    │── open PR ───────────────►│                          │                     │
    │                           │── syntax-check           │                     │
    │                           │── validate ─────────────►│                     │
    │◄── PR ok / failed ────────│                          │                     │
    │                           │                          │                     │
    │── merge to master ───────►│                          │                     │
    │                           │── syntax-check           │                     │
    │                           │── validate ─────────────►│                     │
    │                           │── deploy ───────────────►│ write config        │
    │                           │                          │ restart rsyslog     │
    │                           │── drift-check ──────────►│ compare files       │
    │                           │                          │◄────────────────────│
    │                           │── metric (1 or 0) ───────────────────────────►│
    │                           │                          │                     │
    │                           │  [every 2 min, no push]  │                     │
    │                           │── drift-check ──────────►│ compare files       │
    │                           │── metric (1 or 0) ───────────────────────────►│
    │                           │                          │                     │


Someone edits the server directly (bad):

    rogue admin                 Woodpecker CI              Linux Server          Prometheus
         │                           │                          │                     │
         │── ssh rsyslog-lab         │                          │                     │
         │── vim /etc/rsyslog.conf ──────────────────────────► │ file changed        │
         │                           │                          │                     │
         │                     [2 min later, cron runs]        │                     │
         │                           │── drift-check ──────────►│ diff detected       │
         │                           │── metric 0 (OUT_OF_SYNC)────────────────────►│
         │                           │                          │              alert fires
```

---

## What is the sync metric?

```
gitops_sync_status{repo="rsyslog", server="rsyslog-lab"}
```

| Value | Meaning |
|-------|---------|
| `1`   | Server config matches Git (SYNCED) |
| `0`   | Server config differs from Git (OUT_OF_SYNC) |

Alert on `gitops_sync_status == 0` in Grafana/Alertmanager.

---

## What drift-check actually compares

The drift-check playbook compares files **from the Woodpecker CI container** (always the latest Git commit) against the live server. It checks:

1. `/etc/rsyslog.conf` — must match `files/rsyslog.conf` in Git
2. `/etc/rsyslog.d/30-lab.conf` — must match `files/rsyslog.d/30-lab.conf` in Git
3. Any file managed by Git must not be missing from the server

Files on the server that are **not** in Git (e.g. `50-default.conf`, `20-ufw.conf`) are ignored — they are owned by the OS and are not our concern.

---

## Repository structure

```
.woodpecker.yml              CI pipeline definition
ansible/
  inventory/
    hosts.yml                Server list
    group_vars/all.yml       Variables (paths, user, etc.)
  playbooks/
    validate.yml             Check rsyslog is running and config loads
    apply.yml                Deploy config files from Git to server
    drift-check.yml          Compare Git files to live server (read-only)
files/
  rsyslog.conf               Main rsyslog config (source of truth)
  rsyslog.d/
    30-lab.conf              Drop-in config for lab logging
```

---

## Woodpecker cron setup

Go to **Repository Settings → Crons → Add cron**:

| Field    | Value               |
|----------|---------------------|
| Name     | `gitops_sync_check` |
| Branch   | `master`            |
| Schedule | `*/2 * * * *`       |

---

## Required secrets

Go to **Repository Settings → Secrets**:

| Name              | Description                        |
|-------------------|------------------------------------|
| `SSH_PRIVATE_KEY` | Private key to SSH into the server |