494 lines
15 KiB
Markdown
494 lines
15 KiB
Markdown
# GitOps Status Server Integration - rsyslog Workflow
|
||
|
||
## Overview
|
||
|
||
This document describes the rsyslog repository's integration with **gitops-status-server**, a centralized Kubernetes-based service that manages GitOps status information for multiple repositories.
|
||
|
||
**Architecture:** Rsyslog repo → generates JSON → POST to gitops-status-server → Grafana reads via Infinity datasource
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
```
|
||
┌─ Scheduled Cron (every 2 minutes) ─────────────────────────────┐
|
||
│ │
|
||
│ Woodpecker Trigger: event: cron │
|
||
│ ↓ │
|
||
│ Step: gitops_sync_check │
|
||
│ └─ Run: update-gitops-status.sh │
|
||
│ ├─ Execute: ansible/playbooks/drift-check.yml │
|
||
│ │ └─ Check rsyslog.conf and rsyslog.d/* files │
|
||
│ │ └─ Detect drift vs Git repo │
|
||
│ │ └─ Output: DRIFTED_FILES=file1,file2,file3 │
|
||
│ ├─ Parse: Extract sync status and changed files │
|
||
│ ├─ Generate: JSON payload with metadata │
|
||
│ └─ POST: JSON to gitops-status-server/api/status │
|
||
│ ↓ │
|
||
│ gitops-status-server (Kubernetes service) │
|
||
│ └─ Receives JSON │
|
||
│ └─ Updates internal state │
|
||
│ └─ Serves at: /status.json │
|
||
│ ↓ │
|
||
│ Grafana │
|
||
│ └─ Infinity datasource polls /status.json │
|
||
│ └─ Displays: sync status, drift count, changed files │
|
||
│ │
|
||
└──────────────────────────────────────────────────────────────────┘
|
||
|
||
┌─ Post-Deployment Verification (push to master) ─────────────────┐
|
||
│ │
|
||
│ Woodpecker Pipeline Event: push │
|
||
│ Triggered on: push to master branch │
|
||
│ ↓ │
|
||
│ Steps: syntax-check → validate → deploy → update-gitops-status│
|
||
│ ↓ │
|
||
│ Run: update-gitops-status.sh (same as above) │
|
||
│ └─ Verify deployment succeeded │
|
||
│ └─ Generate JSON status │
|
||
│ └─ POST to gitops-status-server │
|
||
│ ↓ │
|
||
│ gitops-status-server updates /status.json │
|
||
│ ↓ │
|
||
│ Grafana reflects latest deployment status │
|
||
│ │
|
||
└──────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Components
|
||
|
||
### 1. Ansible Playbook: `ansible/playbooks/drift-check.yml`
|
||
|
||
**Purpose:** Detect configuration drift between Git repository and live server
|
||
|
||
**Behavior:**
|
||
- Runs in check mode (read-only, no actual changes)
|
||
- Uses Ansible `copy` module with `check_mode: true`
|
||
- Compares controller files (from Git) with server files
|
||
- Detects:
|
||
- Changes in `/etc/rsyslog.conf`
|
||
- Changes in `/etc/rsyslog.d/*.conf` files
|
||
- Missing files on server
|
||
|
||
**Output (debug messages):**
|
||
- `DRIFTED_FILES=file1,file2,file3` – Comma-separated list of changed files
|
||
- `SYNC_STATUS=SYNCED` or `SYNC_STATUS=OUT_OF_SYNC` – Simple status indicator
|
||
|
||
**Exit Code:**
|
||
- `0` – SYNCED (all tasks pass)
|
||
- `non-zero` – OUT_OF_SYNC (fail task reached)
|
||
|
||
**Example output:**
|
||
```
|
||
DRIFTED_FILES=/etc/rsyslog.conf,/etc/rsyslog.d/30-lab.conf
|
||
SYNC_STATUS=OUT_OF_SYNC
|
||
```
|
||
|
||
---
|
||
|
||
### 2. Shell Script: `update-gitops-status.sh`
|
||
|
||
**Purpose:** Orchestrates the complete status update workflow
|
||
|
||
**Flow:**
|
||
1. **Step 1:** Run `drift-check.yml` playbook
|
||
- Captures stdout/stderr to temp file
|
||
- Stores exit code
|
||
|
||
2. **Step 2:** Parse playbook output
|
||
- Extract `DRIFTED_FILES=...` line
|
||
- Parse comma-separated files
|
||
- Convert full paths to relative paths
|
||
- Determine sync status from exit code
|
||
|
||
3. **Step 3:** Build JSON payload
|
||
- Repo name and server name
|
||
- Sync status (SYNCED / OUT_OF_SYNC)
|
||
- Drift count (number of changed files)
|
||
- Array of changed files with names
|
||
- ISO 8601 timestamp
|
||
|
||
4. **Step 4:** POST JSON to gitops-status-server
|
||
- Endpoint: `$GITOPS_STATUS_SERVER_URL/api/status`
|
||
- Method: POST
|
||
- Content-Type: application/json
|
||
- Check HTTP response code (200-299 = success)
|
||
|
||
**Environment Variables:**
|
||
```bash
|
||
GITOPS_STATUS_SERVER_URL=http://gitops-status-server.observability-stack.svc.cluster.local:80
|
||
REPO_NAME=rsyslog
|
||
SERVER_NAME=rsyslog-lab
|
||
```
|
||
|
||
**JSON Payload Generated:**
|
||
```json
|
||
{
|
||
"repo": "rsyslog",
|
||
"server": "rsyslog-lab",
|
||
"sync_status": "SYNCED",
|
||
"drift_count": 0,
|
||
"files": [],
|
||
"last_check": "2026-04-21T10:30:00Z"
|
||
}
|
||
```
|
||
|
||
**Exit Codes:**
|
||
- `0` – Success (JSON posted to gitops-status-server)
|
||
- `1` – Failure (playbook error, network error, etc.)
|
||
|
||
---
|
||
|
||
### 3. CI/CD Pipeline: `.woodpecker.yml`
|
||
|
||
**Two integration points:**
|
||
|
||
#### a) Post-Deployment (`update-gitops-status` step)
|
||
Runs after successful deployment on push to master:
|
||
|
||
```yaml
|
||
update-gitops-status:
|
||
depends_on: [deploy]
|
||
when:
|
||
branch: master
|
||
event: push
|
||
commands:
|
||
- chmod +x update-gitops-status.sh
|
||
- ./update-gitops-status.sh
|
||
```
|
||
|
||
**Purpose:** Verify deployment and update status immediately
|
||
|
||
#### b) Scheduled Drift Check (`gitops_sync_check` cron)
|
||
Runs on schedule (every 2 minutes by default):
|
||
|
||
```yaml
|
||
gitops_sync_check:
|
||
when:
|
||
event: cron
|
||
commands:
|
||
- chmod +x update-gitops-status.sh
|
||
- ./update-gitops-status.sh
|
||
# Exit with appropriate code so cron shows as success/failed
|
||
- ansible-playbook ... # re-run to determine exit code
|
||
```
|
||
|
||
**Purpose:** Continuous monitoring of server state
|
||
|
||
**Woodpecker UI Setup:**
|
||
1. Go to repository settings
|
||
2. Add cron job:
|
||
- Name: `gitops_sync_check`
|
||
- Branch: `master`
|
||
- Schedule: `*/2 * * * *` (every 2 minutes)
|
||
|
||
---
|
||
|
||
## Data Flow
|
||
|
||
### Input (from Ansible)
|
||
Drift-check playbook outputs structured markers:
|
||
```
|
||
...
|
||
DRIFTED_FILES=/etc/rsyslog.conf,/etc/rsyslog.d/30-lab.conf
|
||
SYNC_STATUS=OUT_OF_SYNC
|
||
...
|
||
```
|
||
|
||
### Processing
|
||
Shell script parses output:
|
||
- Extract drifted files
|
||
- Convert paths: `/etc/rsyslog.conf` → `rsyslog.conf`
|
||
- Count changed files
|
||
- Get current timestamp
|
||
|
||
### Output (to gitops-status-server)
|
||
```json
|
||
{
|
||
"repo": "rsyslog",
|
||
"server": "rsyslog-lab",
|
||
"sync_status": "OUT_OF_SYNC",
|
||
"drift_count": 2,
|
||
"files": [
|
||
{ "name": "rsyslog.conf" },
|
||
{ "name": "rsyslog.d/30-lab.conf" }
|
||
],
|
||
"last_check": "2026-04-21T10:30:00Z"
|
||
}
|
||
```
|
||
|
||
### Final Display (in Grafana)
|
||
Grafana's Infinity datasource reads `/status.json` from gitops-status-server and displays:
|
||
- **Sync Status:** Visual indicator (green ✓ SYNCED / red ✗ OUT_OF_SYNC)
|
||
- **Drift Count:** Number of changed files
|
||
- **Files:** List of which files are different
|
||
- **Last Check:** Timestamp of last drift-check run
|
||
|
||
---
|
||
|
||
## Workflow: Manual Server Changes Detection
|
||
|
||
**Scenario:** Someone manually edits `/etc/rsyslog.conf` directly on the server
|
||
|
||
**Detection flow:**
|
||
1. File is edited on server, Git repo remains unchanged
|
||
2. Cron timer triggers `gitops_sync_check` (every 2 minutes)
|
||
3. Woodpecker runs `update-gitops-status.sh`
|
||
4. Script executes `drift-check.yml` playbook
|
||
5. Ansible copy task in check mode detects difference
|
||
6. Playbook outputs: `DRIFTED_FILES=/etc/rsyslog.conf`
|
||
7. Script parses output and generates JSON
|
||
8. Script POSTs JSON to gitops-status-server
|
||
9. gitops-status-server updates its `/status.json`
|
||
10. Grafana Infinity datasource refreshes
|
||
11. Dashboard shows:
|
||
- Status: OUT_OF_SYNC
|
||
- Drift count: 1
|
||
- Files: [rsyslog.conf]
|
||
- Last check: <current timestamp>
|
||
|
||
**Time to detection:** ≤ 2 minutes (next cron run)
|
||
|
||
---
|
||
|
||
## Workflow: Deployment & Verification
|
||
|
||
**Scenario:** Code is pushed to master branch
|
||
|
||
**Deployment flow:**
|
||
1. Push to master triggers Woodpecker pipeline
|
||
2. Steps: syntax-check → validate → deploy → update-gitops-status
|
||
3. Deploy step runs `ansible/playbooks/apply.yml`
|
||
- Copies Git files to server
|
||
- Makes rsyslog.conf and rsyslog.d/* files match Git
|
||
4. Update-gitops-status step runs:
|
||
- Executes `drift-check.yml` again
|
||
- Should report no drift (files match Git)
|
||
- Generates JSON with `SYNC_STATUS=SYNCED`
|
||
- POSTs JSON to gitops-status-server
|
||
5. gitops-status-server updates `/status.json` to SYNCED
|
||
6. Grafana dashboard immediately shows SYNCED status
|
||
|
||
**Expected result:** After deployment, status should be SYNCED within minutes
|
||
|
||
---
|
||
|
||
## Configuration
|
||
|
||
### Environment Variables
|
||
|
||
Set in `.woodpecker.yml`:
|
||
|
||
```yaml
|
||
environment:
|
||
# URL of gitops-status-server ClusterIP service
|
||
GITOPS_STATUS_SERVER_URL: http://gitops-status-server.observability-stack.svc.cluster.local:80
|
||
|
||
# Repository identifier (for multi-repo dashboards)
|
||
REPO_NAME: rsyslog
|
||
|
||
# Server identifier (for multi-server dashboards)
|
||
SERVER_NAME: rsyslog-lab
|
||
|
||
# SSH private key (for Ansible to connect to server)
|
||
SSH_PRIVATE_KEY: from_secret: SSH_PRIVATE_KEY
|
||
|
||
# Ansible config location
|
||
ANSIBLE_CONFIG: ansible.cfg
|
||
```
|
||
|
||
### Customization
|
||
|
||
To support multiple servers:
|
||
```yaml
|
||
gitops_sync_check:
|
||
commands:
|
||
- bash update-gitops-status.sh # Uses env vars SERVER_NAME
|
||
# Or:
|
||
- SERVER_NAME=rsyslog-lab-1 bash update-gitops-status.sh
|
||
- SERVER_NAME=rsyslog-lab-2 bash update-gitops-status.sh
|
||
```
|
||
|
||
---
|
||
|
||
## Advantages Over Pushgateway
|
||
|
||
| Aspect | Pushgateway | gitops-status-server |
|
||
|--------|-------------|----------------------|
|
||
| **Infrastructure** | Prometheus + Pushgateway | Single Kubernetes service |
|
||
| **Data richness** | Only 0/1 metric | Full JSON with file names, timestamps |
|
||
| **Query language** | PromQL (complex) | Simple JSON API (easy) |
|
||
| **Grafana integration** | Prometheus datasource | Infinity datasource (JSON) |
|
||
| **Multi-repository** | Complex labels | Built-in support |
|
||
| **File-level details** | Not available | Full list of changed files |
|
||
| **Audit trail** | Metrics only | JSON snapshot with metadata |
|
||
|
||
---
|
||
|
||
## Troubleshooting
|
||
|
||
### JSON not being sent to gitops-status-server
|
||
|
||
**Check:**
|
||
1. Verify gitops-status-server is running:
|
||
```bash
|
||
kubectl get pod -n observability-stack | grep gitops-status
|
||
kubectl logs -n observability-stack <pod-name>
|
||
```
|
||
|
||
2. Test connectivity from Woodpecker:
|
||
```bash
|
||
# In Woodpecker log, should see:
|
||
# ==> Sending status to gitops-status-server...
|
||
# ==> Response: HTTP 200
|
||
```
|
||
|
||
3. Check network connectivity:
|
||
```bash
|
||
# From Woodpecker container, test the endpoint:
|
||
curl http://gitops-status-server.observability-stack.svc.cluster.local:80/api/status
|
||
```
|
||
|
||
### Drift not being detected
|
||
|
||
**Check:**
|
||
1. Review drift-check.yml output in Woodpecker logs:
|
||
- Should see "✓ SYNCED" or "✗ OUT OF SYNC"
|
||
- Should see "DRIFTED_FILES=" line
|
||
|
||
2. Manual test on server:
|
||
```bash
|
||
ansible-playbook -i ansible/inventory/hosts.yml ansible/playbooks/drift-check.yml -v
|
||
```
|
||
|
||
3. Check SSH connectivity:
|
||
```bash
|
||
# Verify SSH key is properly set in Woodpecker
|
||
# Check server's rsyslog files are readable:
|
||
ssh user@server ls -la /etc/rsyslog.conf /etc/rsyslog.d/
|
||
```
|
||
|
||
### Cron job not running
|
||
|
||
**Check:**
|
||
1. In Woodpecker UI:
|
||
- Go to repository settings
|
||
- Click "Cron" or look for cron job list
|
||
- Verify `gitops_sync_check` is listed with `*/2 * * * *` schedule
|
||
|
||
2. Check cron execution history:
|
||
- Woodpecker should show execution log
|
||
- Look for "Step 1/4: Running drift-check playbook..."
|
||
|
||
3. Manual trigger:
|
||
- In Woodpecker UI, try to manually trigger the cron job
|
||
- Check logs for errors
|
||
|
||
### Grafana not showing data
|
||
|
||
**Check:**
|
||
1. Verify gitops-status-server is serving JSON:
|
||
```bash
|
||
curl http://gitops-status-server.observability-stack.svc.cluster.local:80/status.json
|
||
```
|
||
|
||
2. Check Grafana Infinity datasource configuration:
|
||
- Datasource URL should point to gitops-status-server
|
||
- Test button should show "Data source is working"
|
||
|
||
3. In dashboard:
|
||
- Edit panel
|
||
- Query should be: `GET /status.json` (or similar)
|
||
- Click "Run query" to test
|
||
- Should see JSON response with data
|
||
|
||
---
|
||
|
||
## Security Considerations
|
||
|
||
- ✓ SSH credentials in Woodpecker secrets (not exposed)
|
||
- ✓ JSON contains only file names and metadata (not config contents)
|
||
- ✓ gitops-status-server only accepts POST from authorized CI/CD
|
||
- ✓ No actual rsyslog config files are exposed
|
||
- ✓ Network communication is internal (ClusterIP)
|
||
|
||
---
|
||
|
||
## Monitoring
|
||
|
||
### Key Metrics
|
||
- **Cron execution:** Every 2 minutes
|
||
- **JSON update frequency:** Every 2 minutes (or after deployment)
|
||
- **Time to detection:** ≤ 2 minutes for manual drift
|
||
- **Time to verification:** Immediate after deployment (post-deploy step)
|
||
|
||
### Grafana Dashboard
|
||
- **Status panel:** Shows SYNCED / OUT_OF_SYNC
|
||
- **Drift count:** Number of changed files
|
||
- **Changed files table:** Lists affected files
|
||
- **Last check:** Timestamp of last check
|
||
- **Repo/Server info:** Identifies which repo/server
|
||
|
||
---
|
||
|
||
## Examples
|
||
|
||
### Example 1: Server is synced with Git
|
||
```json
|
||
{
|
||
"repo": "rsyslog",
|
||
"server": "rsyslog-lab",
|
||
"sync_status": "SYNCED",
|
||
"drift_count": 0,
|
||
"files": [],
|
||
"last_check": "2026-04-21T10:30:00Z"
|
||
}
|
||
```
|
||
|
||
### Example 2: Manual edit on server
|
||
```json
|
||
{
|
||
"repo": "rsyslog",
|
||
"server": "rsyslog-lab",
|
||
"sync_status": "OUT_OF_SYNC",
|
||
"drift_count": 1,
|
||
"files": [
|
||
{ "name": "rsyslog.conf" }
|
||
],
|
||
"last_check": "2026-04-21T10:32:00Z"
|
||
}
|
||
```
|
||
|
||
### Example 3: Multiple files changed
|
||
```json
|
||
{
|
||
"repo": "rsyslog",
|
||
"server": "rsyslog-lab",
|
||
"sync_status": "OUT_OF_SYNC",
|
||
"drift_count": 3,
|
||
"files": [
|
||
{ "name": "rsyslog.conf" },
|
||
{ "name": "rsyslog.d/30-lab.conf" },
|
||
{ "name": "rsyslog.d/31-remote.conf" }
|
||
],
|
||
"last_check": "2026-04-21T10:34:00Z"
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## Summary
|
||
|
||
This integration provides:
|
||
- ✓ Real-time drift detection via cron (every 2 minutes)
|
||
- ✓ Post-deployment verification
|
||
- ✓ File-level granularity (which files changed)
|
||
- ✓ Integration with gitops-status-server (centralized service)
|
||
- ✓ Grafana Infinity datasource support
|
||
- ✓ Clean JSON-based architecture
|
||
- ✓ No Pushgateway dependency for this use case
|
||
|
||
The rsyslog repository is now responsible for producing clean, structured JSON snapshots of its GitOps status, which are consumed by gitops-status-server and displayed in Grafana.
|