rsyslog/DEPLOYMENT_CHECKLIST.md

Deployment Checklist

This checklist guides you through deploying the updated rsyslog repository with gitops-status-server integration.

Prerequisites

Before deploying, ensure:

• gitops-status-server is deployed and accessible at:
  http://gitops-status-server.observability-stack.svc.cluster.local:80
• gitops-status-server has /api/status endpoint implemented (see GITOPS_STATUS_API_REFERENCE.md)
• Woodpecker CI is configured for this repository
• SSH access to rsyslog-lab server is configured
• SSH_PRIVATE_KEY secret is set in Woodpecker repository settings

Files Changed/Added

New Files

• ✅ update-gitops-status.sh - Main script for JSON status generation
• ✅ GITOPS_STATUS_INTEGRATION.md - Integration documentation
• ✅ GITOPS_STATUS_API_REFERENCE.md - API reference with examples
• ✅ MIGRATION_SUMMARY.md - Summary of all changes made

Modified Files

• ✅ .woodpecker.yml - Updated pipeline to use JSON status
• ✅ README.md - Updated documentation with new flow

Unchanged Files (no action needed)

• ✅ All Ansible playbooks
• ✅ All Ansible inventory files
• ✅ All rsyslog config files
• ✅ Local scripts (apply.sh, drift-check.sh)

Deployment Steps

1. Review Changes

• Read MIGRATION_SUMMARY.md to understand all changes
• Review .woodpecker.yml pipeline changes
• Review update-gitops-status.sh script logic

2. Verify gitops-status-server

Test the gitops-status-server API endpoint:

# Test POST endpoint (should return 200 or 404 if not implemented yet)
curl -X POST http://gitops-status-server.observability-stack.svc.cluster.local:80/api/status \
  -H "Content-Type: application/json" \
  -d '{
    "repo": "test",
    "server": "test",
    "sync_status": "SYNCED",
    "drift_count": 0,
    "files": [],
    "last_check": "2026-04-21T10:00:00Z"
  }'

# Test GET endpoint
curl http://gitops-status-server.observability-stack.svc.cluster.local:80/status.json

If the API is not implemented yet:

• Implement gitops-status-server API (use GITOPS_STATUS_API_REFERENCE.md)
• Deploy to Kubernetes cluster
• Verify endpoints are accessible

3. Test Locally (Optional)

Before pushing to Git, you can test the script locally:

# Set environment variables
export GITOPS_STATUS_SERVER_URL="http://gitops-status-server.observability-stack.svc.cluster.local:80"
export REPO_NAME="rsyslog"
export SERVER_NAME="rsyslog-lab"

# Make script executable
chmod +x update-gitops-status.sh

# Run the script (requires Ansible, jq, curl)
./update-gitops-status.sh

Expected output:

==> Running drift check playbook...
    Inventory: ansible/inventory/hosts.yml
    Playbook:  ansible/playbooks/drift-check.yml
==> Status: SYNCED - server configuration matches Git
==> Drift count: 0
==> Generated JSON status:
{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "SYNCED",
  "drift_count": 0,
  "files": [],
  "last_check": "2026-04-21T10:30:00Z"
}
==> Sending status to gitops-status-server...
    URL: http://gitops-status-server.observability-stack.svc.cluster.local:80/api/status
==> Status update successful (HTTP 200)

4. Commit and Push Changes

# Stage all changes
git add .

# Commit
git commit -m "Migrate from Pushgateway to gitops-status-server JSON status

- Add update-gitops-status.sh script for JSON status generation
- Update .woodpecker.yml to use gitops-status-server
- Remove Pushgateway metric push logic
- Add comprehensive documentation
- Keep all Ansible playbooks unchanged"

# Push to master (will trigger deploy pipeline)
git push origin master

5. Monitor First Deployment

After pushing to master:

• Watch Woodpecker pipeline execution
• Verify syntax-check step passes
• Verify validate step passes
• Verify deploy step completes
• Verify update-gitops-status step runs successfully
• Check that JSON is sent to gitops-status-server

Example successful update-gitops-status step output:

==> Running post-deploy GitOps status check...
==> Running drift check playbook...
==> Status: SYNCED - server configuration matches Git
==> Drift count: 0
==> Generated JSON status: {...}
==> Sending status to gitops-status-server...
==> Status update successful (HTTP 200)
==> JSON status update complete. Pipeline always succeeds.

6. Verify Cron Pipeline

The cron pipeline runs every 2 minutes:

• Wait for next cron execution
• Check Woodpecker for gitops_sync_check pipeline run
• Verify JSON status is sent
• Verify pipeline succeeds (if synced) or fails (if drift detected)

7. Test Drift Detection

Manually create drift to test detection:

# SSH to the server
ssh rsyslog-lab

# Edit a config file
echo "# manual edit" >> /etc/rsyslog.conf

# Wait up to 2 minutes for next cron run

Expected behavior:

• Cron pipeline runs
• Drift detected in Ansible playbook
• JSON sent with sync_status: "OUT_OF_SYNC"
• JSON includes files: [{"name": "rsyslog.conf"}]
• Pipeline marked as FAILED (for visibility)

Verify in gitops-status-server:

curl http://gitops-status-server.observability-stack.svc.cluster.local:80/status.json

Should show:

[
  {
    "repo": "rsyslog",
    "server": "rsyslog-lab",
    "sync_status": "OUT_OF_SYNC",
    "drift_count": 1,
    "files": [
      {"name": "rsyslog.conf"}
    ],
    "last_check": "...",
    "updated_at": "..."
  }
]

8. Configure Grafana Dashboard

• Add Infinity datasource pointing to gitops-status-server
• Create dashboard to display GitOps status
• Add panels for:
  • Sync status overview (SYNCED vs OUT_OF_SYNC)
  • Drift count per repo
  • Detailed file list for drifted repos
  • Last check timestamp
  • Historical trend (if gitops-status-server stores history)

Example dashboard queries provided in GITOPS_STATUS_API_REFERENCE.md.

9. Cleanup (Optional)

If everything works correctly:

• Remove old Pushgateway metrics for rsyslog (if no longer needed)
• Update any alerts/dashboards that used old gitops_sync_status metric
• Document the new JSON status format in team wiki/docs

Troubleshooting

Script fails with "command not found: jq"

Problem: jq is not installed in the Woodpecker container

Solution: The .woodpecker.yml already includes apk add --no-cache jq. Verify the step runs before the script.

Script fails with "HTTP 404" or "HTTP 500"

Problem: gitops-status-server endpoint not implemented or not accessible

Solution:

1. Verify gitops-status-server is running: curl http://gitops-status-server.observability-stack.svc.cluster.local:80/health
2. Check Kubernetes service: kubectl get svc -n observability-stack
3. Implement /api/status endpoint using GITOPS_STATUS_API_REFERENCE.md

Script fails with "No such file or directory: update-gitops-status.sh"

Problem: Script not found in workspace

Solution: The script is created in the repository root. Verify it's committed to Git and available in the CI workspace.

Drift not detected when expected

Problem: Manual changes not showing up in drift check

Solution:

1. Verify changes are to files managed by Git (rsyslog.conf or rsyslog.d/*.conf)
2. Check Ansible playbook output for diff details
3. Verify SSH access to server from CI container

JSON has empty files array even when drift detected

Problem: Script parsing logic not extracting filenames correctly

Solution:

1. Check Ansible output format - script expects specific JSON structure
2. Run with ANSIBLE_STDOUT_CALLBACK=json to see raw output
3. Update regex patterns in update-gitops-status.sh if needed

Rollback Procedure

If you need to rollback to Pushgateway:

# Revert to previous commit
git revert HEAD

# Or restore specific files
git checkout HEAD~1 .woodpecker.yml
git checkout HEAD~1 README.md

# Remove new files
git rm update-gitops-status.sh GITOPS_STATUS_*.md MIGRATION_SUMMARY.md

# Commit and push
git commit -m "Rollback to Pushgateway metrics"
git push origin master

Success Criteria

✅ All checks passed when:

• Woodpecker pipeline completes successfully on push to master
• JSON status is sent to gitops-status-server after deploy
• Cron pipeline runs every 2 minutes and sends JSON status
• Drift is correctly detected and reported in JSON
• gitops-status-server /status.json endpoint returns correct data
• Grafana dashboard displays rsyslog sync status
• No errors in Woodpecker logs
• File-level drift details are visible in Grafana

Additional Resources

• README.md - Repository overview and workflow
• MIGRATION_SUMMARY.md - Detailed migration changes
• GITOPS_STATUS_INTEGRATION.md - Integration architecture
• GITOPS_STATUS_API_REFERENCE.md - API implementation guide

Support

If you encounter issues:

1. Check Woodpecker pipeline logs
2. Verify gitops-status-server logs
3. Test API endpoints manually with curl
4. Review Ansible playbook output
5. Check this repository's documentation files