rsyslog/README_GITOPS_STATUS.md

# rsyslog GitOps: gitops-status-server Integration

## Overview

This repository now uses **gitops-status-server** (a Kubernetes-based service) for GitOps status monitoring, replacing Prometheus Pushgateway.

**Status:** ✅ Ready for deployment

---

## What This Means

### For You (Operator)
- Configure Woodpecker cron job once (5 minutes)
- Cron runs every 2 minutes and checks if rsyslog config on server matches Git
- If changes detected, JSON is automatically sent to gitops-status-server
- Grafana shows:
  - **Sync status** (green=SYNCED, red=OUT_OF_SYNC)
  - **Drift count** (how many files changed)
  - **Changed files** (list of which files are different)
  - **Last check** (when drift was last checked)

### For Grafana
- Connects to gitops-status-server via Infinity datasource
- Reads JSON status snapshots
- Displays rich metadata about rsyslog configuration state

### For the System
- Detects manual edits on the server within 2 minutes
- Verifies deployment succeeded immediately after push
- No Pushgateway required
- Clean JSON-based architecture

---

## Quick Start (5 Steps)

### Step 1: Verify Files Are In Place
```bash
# Check these files exist:
ls -la .woodpecker.yml                                 # ✓ Updated
ls -la ansible/playbooks/drift-check.yml              # ✓ Enhanced
ls -la update-gitops-status.sh                        # ✓ New script
```

### Step 2: Make Script Executable
```bash
chmod +x update-gitops-status.sh
git add update-gitops-status.sh
git commit -m "add: gitops-status-server integration script"
git push
```

### Step 3: Verify Pipeline Runs
- Push should trigger Woodpecker pipeline
- New step `update-gitops-status` should execute after `deploy`
- Check logs for: "✓ Status update successful"

### Step 4: Configure Woodpecker Cron Job
In Woodpecker UI:
1. Go to repository
2. Click **Settings** → **Cron**
3. Click **Add Cron**
4. Fill in:
   - **Name:** `gitops_sync_check`
   - **Branch:** `master`
   - **Schedule:** `*/2 * * * *`
5. Click **Save**

### Step 5: Test Cron Job
- Wait max 2 minutes for cron to trigger
- Check Woodpecker logs for cron execution
- Verify gitops-status-server received POST:
  ```bash
  kubectl logs -n observability-stack -l app=gitops-status-server | grep POST
  ```

---

## How It Works

### Architecture Diagram
```
Every 2 minutes (or after deployment):

Woodpecker
├─ gitops_sync_check or update-gitops-status step
└─ Runs: update-gitops-status.sh
   ├─ 1. Execute: ansible/playbooks/drift-check.yml
   │  └─ Compare Git files vs server files (read-only)
   │  └─ Output: DRIFTED_FILES=file1,file2,file3
   ├─ 2. Parse: Extract changed files and sync status
   ├─ 3. Generate: JSON payload with metadata
   └─ 4. POST: JSON to gitops-status-server/api/status

gitops-status-server (K8s Service)
└─ Receives JSON
└─ Updates /status.json
└─ Serves via HTTP

Grafana
└─ Infinity datasource
└─ Polls /status.json (periodic refresh)
└─ Displays in dashboard panel
```

### Data Flow

**Input:** Ansible drift-check output
```
DRIFTED_FILES=/etc/rsyslog.conf,/etc/rsyslog.d/30-lab.conf
SYNC_STATUS=OUT_OF_SYNC
```

**Output:** JSON sent to gitops-status-server
```json
{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "OUT_OF_SYNC",
  "drift_count": 2,
  "files": [
    { "name": "rsyslog.conf" },
    { "name": "rsyslog.d/30-lab.conf" }
  ],
  "last_check": "2026-04-21T10:30:00Z"
}
```

**Display:** Grafana dashboard
- 🔴 OUT_OF_SYNC (red card)
- Drift count: 2
- Files: rsyslog.conf, rsyslog.d/30-lab.conf
- Last check: 2026-04-21 10:30 UTC

---

## Files Changed

### `.woodpecker.yml`
**Changes:**
- Renamed step: `update-sync-metric` → `update-gitops-status`
- Changed command: replaced Pushgateway push → `update-gitops-status.sh` script call
- Added environment variables:
  - `GITOPS_STATUS_SERVER_URL`
  - `REPO_NAME`
  - `SERVER_NAME`
- Both `update-gitops-status` (post-deploy) and `gitops_sync_check` (cron) now use the script

### `ansible/playbooks/drift-check.yml`
**Changes:**
- Added file collection: builds list of changed files in `drifted_files` fact
- Added debug output: prints `DRIFTED_FILES=file1,file2,file3` for script parsing
- Added status markers: prints `SYNC_STATUS=SYNCED` or `SYNC_STATUS=OUT_OF_SYNC`
- **No changes to drift detection logic** (fully backward compatible)

### `update-gitops-status.sh` (NEW)
**Purpose:** Orchestrates status generation and delivery
**4 Steps:**
1. Run drift-check.yml
2. Parse output to extract changed files
3. Build JSON payload
4. POST to gitops-status-server/api/status

---

## Testing

### Test 1: Verify script works locally
```bash
# From repo root, with SSH key configured
./update-gitops-status.sh

# Expected output:
# ═══════════════════════════════════════════════════
# Step 1/4: Running drift-check playbook...
# [playbook output...]
# Step 2/4: Analyzing drift detection results...
# Step 3/4: Building JSON payload...
# Generated JSON:
# {
#   "repo": "rsyslog",
#   ...
# }
# Step 4/4: Sending status to gitops-status-server...
# Response: HTTP 200
# ✓ Status update successful
```

### Test 2: Verify Woodpecker pipeline runs
1. Make a change to a file
2. Push to master branch
3. Woodpecker pipeline should:
   - Run syntax-check ✓
   - Run validate ✓
   - Run deploy ✓
   - Run update-gitops-status ✓
4. Check logs for: "✓ Status update successful"

### Test 3: Verify cron job triggers
1. Woodpecker cron job configured for `*/2 * * * *`
2. Wait 2 minutes
3. Check Woodpecker UI for cron execution
4. Check logs for drift-check output

### Test 4: Verify gitops-status-server receives JSON
```bash
# Check gitops-status-server logs
kubectl logs -n observability-stack -l app=gitops-status-server -f

# Should show POST requests:
# POST /api/status - 200 OK
```

### Test 5: Verify Grafana dashboard
1. Open Grafana
2. Check Infinity datasource:
   - Should show "Data source is working"
3. Check dashboard panel:
   - Should display sync status
   - Should show drift count
   - Should list changed files (if any)

---

## Troubleshooting

### Issue: Cron job not running
**Check:**
1. Is cron job configured in Woodpecker?
   - Go to repo settings → Cron
   - Should see `gitops_sync_check` with `*/2 * * * *` schedule
2. Is the schedule active?
   - Cron should have triggered at least once

**Fix:**
- Create cron job if missing
- Verify schedule is `*/2 * * * *`
- Check branch is `master`

### Issue: "HTTP 500" or "HTTP 503" when posting
**Check:**
1. Is gitops-status-server running?
   ```bash
   kubectl get pod -n observability-stack | grep gitops-status
   ```
2. Is it ready?
   ```bash
   kubectl get pod -n observability-stack -o wide | grep gitops-status
   ```
3. Check logs:
   ```bash
   kubectl logs -n observability-stack -l app=gitops-status-server
   ```

**Fix:**
- Restart gitops-status-server if needed
- Check error logs for API issues
- Verify /api/status endpoint exists

### Issue: Drift not detected
**Check:**
1. Run drift-check manually:
   ```bash
   ansible-playbook -i ansible/inventory/hosts.yml ansible/playbooks/drift-check.yml -v
   ```
2. Does it report the correct status?
3. SSH connectivity to server?

**Fix:**
- Check SSH key is set in Woodpecker secrets
- Verify server files are readable: `ssh user@server ls -la /etc/rsyslog.conf`
- Check Ansible inventory is correct

### Issue: JSON not sent (HTTP 000)
**Check:**
1. Is gitops-status-server URL correct?
   ```bash
   curl http://gitops-status-server.observability-stack.svc.cluster.local:80
   ```
2. Can Woodpecker reach it?
   - May be network/DNS issue

**Fix:**
- Check gitops-status-server hostname/port
- Test from Woodpecker container: `curl http://gitops-status-server:80`
- Check Woodpecker network policies

### Issue: Grafana shows "No data"
**Check:**
1. Does Infinity datasource work?
   - Go to Data Sources → test
2. Can Grafana reach gitops-status-server?
3. What query is the panel using?

**Fix:**
- Verify datasource URL is correct
- Check query in panel: should be `/status.json` or similar
- Ensure gitops-status-server is returning JSON

---

## Examples

### Example 1: Server is synced
```json
{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "SYNCED",
  "drift_count": 0,
  "files": [],
  "last_check": "2026-04-21T10:32:00Z"
}
```

**Grafana display:**
- 🟢 SYNCED
- Drift: 0
- Files: (empty)

### Example 2: Manual edit on server
```json
{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "OUT_OF_SYNC",
  "drift_count": 1,
  "files": [
    { "name": "rsyslog.conf" }
  ],
  "last_check": "2026-04-21T10:34:00Z"
}
```

**Grafana display:**
- 🔴 OUT OF SYNC
- Drift: 1
- Files: rsyslog.conf

### Example 3: Multiple files changed after deployment
```json
{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "SYNCED",
  "drift_count": 0,
  "files": [],
  "last_check": "2026-04-21T10:36:00Z"
}
```

**Timeline:**
1. 10:30 - Push to master triggers deploy
2. 10:31 - Deploy completes, files changed
3. 10:31 - update-gitops-status runs, verifies sync
4. 10:31 - JSON sent: SYNCED
5. 10:36 - Grafana shows ✓ SYNCED (5 min later)

---

## Environment Configuration

The following environment variables are set in `.woodpecker.yml`:

```yaml
GITOPS_STATUS_SERVER_URL: http://gitops-status-server.observability-stack.svc.cluster.local:80
REPO_NAME: rsyslog
SERVER_NAME: rsyslog-lab
SSH_PRIVATE_KEY: from_secret: SSH_PRIVATE_KEY
ANSIBLE_CONFIG: ansible.cfg
```

**To customize:**
- Edit `.woodpecker.yml`
- Change environment variables under `update-gitops-status` and `gitops_sync_check` steps
- Push and re-run

---

## Key Features

✅ **Automatic drift detection** – Every 2 minutes  
✅ **Post-deployment verification** – Immediate after deploy  
✅ **File-level details** – Shows which files changed  
✅ **No Pushgateway** – Simplified infrastructure  
✅ **Grafana integration** – Infinity datasource (native)  
✅ **Audit trail** – JSON snapshots with timestamps  
✅ **Multi-server ready** – Structured for scale  

---

## Documentation

| Document | Purpose |
|----------|---------|
| `GITOPS_STATUS_SERVER_INTEGRATION.md` | Comprehensive architecture & flow |
| `QUICK_REFERENCE.md` | Quick start & troubleshooting |
| `REFACTOR_SUMMARY.md` | Before/after comparison |
| This file | Overview & quick start |

---

## Next Steps

1. **Verify files are in place:**
   ```bash
   git status
   ```

2. **Push changes:**
   ```bash
   git push
   ```

3. **Monitor first pipeline run:**
   - Check Woodpecker logs
   - Look for `update-gitops-status` step
   - Verify HTTP 200 response

4. **Configure cron job:**
   - Go to Woodpecker UI
   - Add cron: `gitops_sync_check` at `*/2 * * * *`

5. **Test cron execution:**
   - Wait 2 minutes
   - Check Woodpecker logs
   - Verify gitops-status-server receives JSON

6. **Verify Grafana:**
   - Check dashboard displays sync status
   - Test with manual file edit on server
   - Verify detection within 2 minutes

---

## Support

For issues or questions:
1. Check `QUICK_REFERENCE.md` troubleshooting section
2. Review Woodpecker pipeline logs
3. Check gitops-status-server application logs:
   ```bash
   kubectl logs -n observability-stack -l app=gitops-status-server -f
   ```
4. Test connectivity manually:
   ```bash
   curl http://gitops-status-server.observability-stack.svc.cluster.local:80/api/status
   ```

---

## Summary

You now have a clean, production-ready GitOps status monitoring system that:
- Detects configuration drift every 2 minutes
- Sends rich metadata (file names, timestamps) to gitops-status-server
- Integrates with Grafana via Infinity datasource
- Requires minimal infrastructure (no Pushgateway)
- Works reliably for multi-server deployments

**Status:** ✅ Ready to deploy and use