# GitOps Status Server Integration - rsyslog Workflow

## Overview

This document describes the rsyslog repository's integration with **gitops-status-server**, a centralized Kubernetes-based service that manages GitOps status information for multiple repositories.

**Architecture:** Rsyslog repo → generates JSON → POST to gitops-status-server → Grafana reads via Infinity datasource

---

## Architecture

```
┌─ Scheduled Cron (every 2 minutes) ─────────────────────────────┐
│                                                                  │
│  Woodpecker Trigger: event: cron                               │
│        ↓                                                        │
│  Step: gitops_sync_check                                       │
│  └─ Run: update-gitops-status.sh                              │
│     ├─ Execute: ansible/playbooks/drift-check.yml             │
│     │  └─ Check rsyslog.conf and rsyslog.d/* files           │
│     │  └─ Detect drift vs Git repo                             │
│     │  └─ Output: DRIFTED_FILES=file1,file2,file3            │
│     ├─ Parse: Extract sync status and changed files          │
│     ├─ Generate: JSON payload with metadata                   │
│     └─ POST: JSON to gitops-status-server/api/status         │
│        ↓                                                        │
│  gitops-status-server (Kubernetes service)                     │
│  └─ Receives JSON                                              │
│  └─ Updates internal state                                     │
│  └─ Serves at: /status.json                                   │
│        ↓                                                        │
│  Grafana                                                        │
│  └─ Infinity datasource polls /status.json                    │
│  └─ Displays: sync status, drift count, changed files        │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘

┌─ Post-Deployment Verification (push to master) ─────────────────┐
│                                                                  │
│  Woodpecker Pipeline Event: push                               │
│  Triggered on: push to master branch                           │
│        ↓                                                        │
│  Steps: syntax-check → validate → deploy → update-gitops-status│
│                                                  ↓               │
│  Run: update-gitops-status.sh (same as above)                 │
│  └─ Verify deployment succeeded                               │
│  └─ Generate JSON status                                       │
│  └─ POST to gitops-status-server                              │
│        ↓                                                        │
│  gitops-status-server updates /status.json                     │
│        ↓                                                        │
│  Grafana reflects latest deployment status                     │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘
```

---

## Components

### 1. Ansible Playbook: `ansible/playbooks/drift-check.yml`

**Purpose:** Detect configuration drift between Git repository and live server

**Behavior:**
- Runs in check mode (read-only, no actual changes)
- Uses Ansible `copy` module with `check_mode: true`
- Compares controller files (from Git) with server files
- Detects:
  - Changes in `/etc/rsyslog.conf`
  - Changes in `/etc/rsyslog.d/*.conf` files
  - Missing files on server

**Output (debug messages):**
- `DRIFTED_FILES=file1,file2,file3` – Comma-separated list of changed files
- `SYNC_STATUS=SYNCED` or `SYNC_STATUS=OUT_OF_SYNC` – Simple status indicator

**Exit Code:**
- `0` – SYNCED (all tasks pass)
- `non-zero` – OUT_OF_SYNC (fail task reached)

**Example output:**
```
DRIFTED_FILES=/etc/rsyslog.conf,/etc/rsyslog.d/30-lab.conf
SYNC_STATUS=OUT_OF_SYNC
```

---

### 2. Shell Script: `update-gitops-status.sh`

**Purpose:** Orchestrates the complete status update workflow

**Flow:**
1. **Step 1:** Run `drift-check.yml` playbook
   - Captures stdout/stderr to temp file
   - Stores exit code

2. **Step 2:** Parse playbook output
   - Extract `DRIFTED_FILES=...` line
   - Parse comma-separated files
   - Convert full paths to relative paths
   - Determine sync status from exit code

3. **Step 3:** Build JSON payload
   - Repo name and server name
   - Sync status (SYNCED / OUT_OF_SYNC)
   - Drift count (number of changed files)
   - Array of changed files with names
   - ISO 8601 timestamp

4. **Step 4:** POST JSON to gitops-status-server
   - Endpoint: `$GITOPS_STATUS_SERVER_URL/api/status`
   - Method: POST
   - Content-Type: application/json
   - Check HTTP response code (200-299 = success)

**Environment Variables:**
```bash
GITOPS_STATUS_SERVER_URL=http://gitops-status-server.observability-stack.svc.cluster.local:80
REPO_NAME=rsyslog
SERVER_NAME=rsyslog-lab
```

**JSON Payload Generated:**
```json
{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "SYNCED",
  "drift_count": 0,
  "files": [],
  "last_check": "2026-04-21T10:30:00Z"
}
```

**Exit Codes:**
- `0` – Success (JSON posted to gitops-status-server)
- `1` – Failure (playbook error, network error, etc.)

---

### 3. CI/CD Pipeline: `.woodpecker.yml`

**Two integration points:**

#### a) Post-Deployment (`update-gitops-status` step)
Runs after successful deployment on push to master:

```yaml
update-gitops-status:
  depends_on: [deploy]
  when:
    branch: master
    event: push
  commands:
    - chmod +x update-gitops-status.sh
    - ./update-gitops-status.sh
```

**Purpose:** Verify deployment and update status immediately

#### b) Scheduled Drift Check (`gitops_sync_check` cron)
Runs on schedule (every 2 minutes by default):

```yaml
gitops_sync_check:
  when:
    event: cron
  commands:
    - chmod +x update-gitops-status.sh
    - ./update-gitops-status.sh
    # Exit with appropriate code so cron shows as success/failed
    - ansible-playbook ... # re-run to determine exit code
```

**Purpose:** Continuous monitoring of server state

**Woodpecker UI Setup:**
1. Go to repository settings
2. Add cron job:
   - Name: `gitops_sync_check`
   - Branch: `master`
   - Schedule: `*/2 * * * *` (every 2 minutes)

---

## Data Flow

### Input (from Ansible)
Drift-check playbook outputs structured markers:
```
...
DRIFTED_FILES=/etc/rsyslog.conf,/etc/rsyslog.d/30-lab.conf
SYNC_STATUS=OUT_OF_SYNC
...
```

### Processing
Shell script parses output:
- Extract drifted files
- Convert paths: `/etc/rsyslog.conf` → `rsyslog.conf`
- Count changed files
- Get current timestamp

### Output (to gitops-status-server)
```json
{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "OUT_OF_SYNC",
  "drift_count": 2,
  "files": [
    { "name": "rsyslog.conf" },
    { "name": "rsyslog.d/30-lab.conf" }
  ],
  "last_check": "2026-04-21T10:30:00Z"
}
```

### Final Display (in Grafana)
Grafana's Infinity datasource reads `/status.json` from gitops-status-server and displays:
- **Sync Status:** Visual indicator (green ✓ SYNCED / red ✗ OUT_OF_SYNC)
- **Drift Count:** Number of changed files
- **Files:** List of which files are different
- **Last Check:** Timestamp of last drift-check run

---

## Workflow: Manual Server Changes Detection

**Scenario:** Someone manually edits `/etc/rsyslog.conf` directly on the server

**Detection flow:**
1. File is edited on server, Git repo remains unchanged
2. Cron timer triggers `gitops_sync_check` (every 2 minutes)
3. Woodpecker runs `update-gitops-status.sh`
4. Script executes `drift-check.yml` playbook
5. Ansible copy task in check mode detects difference
6. Playbook outputs: `DRIFTED_FILES=/etc/rsyslog.conf`
7. Script parses output and generates JSON
8. Script POSTs JSON to gitops-status-server
9. gitops-status-server updates its `/status.json`
10. Grafana Infinity datasource refreshes
11. Dashboard shows:
    - Status: OUT_OF_SYNC
    - Drift count: 1
    - Files: [rsyslog.conf]
    - Last check: <current timestamp>

**Time to detection:** ≤ 2 minutes (next cron run)

---

## Workflow: Deployment & Verification

**Scenario:** Code is pushed to master branch

**Deployment flow:**
1. Push to master triggers Woodpecker pipeline
2. Steps: syntax-check → validate → deploy → update-gitops-status
3. Deploy step runs `ansible/playbooks/apply.yml`
   - Copies Git files to server
   - Makes rsyslog.conf and rsyslog.d/* files match Git
4. Update-gitops-status step runs:
   - Executes `drift-check.yml` again
   - Should report no drift (files match Git)
   - Generates JSON with `SYNC_STATUS=SYNCED`
   - POSTs JSON to gitops-status-server
5. gitops-status-server updates `/status.json` to SYNCED
6. Grafana dashboard immediately shows SYNCED status

**Expected result:** After deployment, status should be SYNCED within minutes

---

## Configuration

### Environment Variables

Set in `.woodpecker.yml`:

```yaml
environment:
  # URL of gitops-status-server ClusterIP service
  GITOPS_STATUS_SERVER_URL: http://gitops-status-server.observability-stack.svc.cluster.local:80
  
  # Repository identifier (for multi-repo dashboards)
  REPO_NAME: rsyslog
  
  # Server identifier (for multi-server dashboards)
  SERVER_NAME: rsyslog-lab
  
  # SSH private key (for Ansible to connect to server)
  SSH_PRIVATE_KEY: from_secret: SSH_PRIVATE_KEY
  
  # Ansible config location
  ANSIBLE_CONFIG: ansible.cfg
```

### Customization

To support multiple servers:
```yaml
gitops_sync_check:
  commands:
    - bash update-gitops-status.sh  # Uses env vars SERVER_NAME
    # Or:
    - SERVER_NAME=rsyslog-lab-1 bash update-gitops-status.sh
    - SERVER_NAME=rsyslog-lab-2 bash update-gitops-status.sh
```

---

## Advantages Over Pushgateway

| Aspect | Pushgateway | gitops-status-server |
|--------|-------------|----------------------|
| **Infrastructure** | Prometheus + Pushgateway | Single Kubernetes service |
| **Data richness** | Only 0/1 metric | Full JSON with file names, timestamps |
| **Query language** | PromQL (complex) | Simple JSON API (easy) |
| **Grafana integration** | Prometheus datasource | Infinity datasource (JSON) |
| **Multi-repository** | Complex labels | Built-in support |
| **File-level details** | Not available | Full list of changed files |
| **Audit trail** | Metrics only | JSON snapshot with metadata |

---

## Troubleshooting

### JSON not being sent to gitops-status-server

**Check:**
1. Verify gitops-status-server is running:
   ```bash
   kubectl get pod -n observability-stack | grep gitops-status
   kubectl logs -n observability-stack <pod-name>
   ```

2. Test connectivity from Woodpecker:
   ```bash
   # In Woodpecker log, should see:
   # ==> Sending status to gitops-status-server...
   # ==> Response: HTTP 200
   ```

3. Check network connectivity:
   ```bash
   # From Woodpecker container, test the endpoint:
   curl http://gitops-status-server.observability-stack.svc.cluster.local:80/api/status
   ```

### Drift not being detected

**Check:**
1. Review drift-check.yml output in Woodpecker logs:
   - Should see "✓ SYNCED" or "✗ OUT OF SYNC"
   - Should see "DRIFTED_FILES=" line

2. Manual test on server:
   ```bash
   ansible-playbook -i ansible/inventory/hosts.yml ansible/playbooks/drift-check.yml -v
   ```

3. Check SSH connectivity:
   ```bash
   # Verify SSH key is properly set in Woodpecker
   # Check server's rsyslog files are readable:
   ssh user@server ls -la /etc/rsyslog.conf /etc/rsyslog.d/
   ```

### Cron job not running

**Check:**
1. In Woodpecker UI:
   - Go to repository settings
   - Click "Cron" or look for cron job list
   - Verify `gitops_sync_check` is listed with `*/2 * * * *` schedule

2. Check cron execution history:
   - Woodpecker should show execution log
   - Look for "Step 1/4: Running drift-check playbook..."

3. Manual trigger:
   - In Woodpecker UI, try to manually trigger the cron job
   - Check logs for errors

### Grafana not showing data

**Check:**
1. Verify gitops-status-server is serving JSON:
   ```bash
   curl http://gitops-status-server.observability-stack.svc.cluster.local:80/status.json
   ```

2. Check Grafana Infinity datasource configuration:
   - Datasource URL should point to gitops-status-server
   - Test button should show "Data source is working"

3. In dashboard:
   - Edit panel
   - Query should be: `GET /status.json` (or similar)
   - Click "Run query" to test
   - Should see JSON response with data

---

## Security Considerations

- ✓ SSH credentials in Woodpecker secrets (not exposed)
- ✓ JSON contains only file names and metadata (not config contents)
- ✓ gitops-status-server only accepts POST from authorized CI/CD
- ✓ No actual rsyslog config files are exposed
- ✓ Network communication is internal (ClusterIP)

---

## Monitoring

### Key Metrics
- **Cron execution:** Every 2 minutes
- **JSON update frequency:** Every 2 minutes (or after deployment)
- **Time to detection:** ≤ 2 minutes for manual drift
- **Time to verification:** Immediate after deployment (post-deploy step)

### Grafana Dashboard
- **Status panel:** Shows SYNCED / OUT_OF_SYNC
- **Drift count:** Number of changed files
- **Changed files table:** Lists affected files
- **Last check:** Timestamp of last check
- **Repo/Server info:** Identifies which repo/server

---

## Examples

### Example 1: Server is synced with Git
```json
{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "SYNCED",
  "drift_count": 0,
  "files": [],
  "last_check": "2026-04-21T10:30:00Z"
}
```

### Example 2: Manual edit on server
```json
{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "OUT_OF_SYNC",
  "drift_count": 1,
  "files": [
    { "name": "rsyslog.conf" }
  ],
  "last_check": "2026-04-21T10:32:00Z"
}
```

### Example 3: Multiple files changed
```json
{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "OUT_OF_SYNC",
  "drift_count": 3,
  "files": [
    { "name": "rsyslog.conf" },
    { "name": "rsyslog.d/30-lab.conf" },
    { "name": "rsyslog.d/31-remote.conf" }
  ],
  "last_check": "2026-04-21T10:34:00Z"
}
```

---

## Summary

This integration provides:
- ✓ Real-time drift detection via cron (every 2 minutes)
- ✓ Post-deployment verification
- ✓ File-level granularity (which files changed)
- ✓ Integration with gitops-status-server (centralized service)
- ✓ Grafana Infinity datasource support
- ✓ Clean JSON-based architecture
- ✓ No Pushgateway dependency for this use case

The rsyslog repository is now responsible for producing clean, structured JSON snapshots of its GitOps status, which are consumed by gitops-status-server and displayed in Grafana.