rsyslog/DEBUGGING_GITOPS_STATUS.md

Debugging GitOps Status Issues

This guide helps troubleshoot issues with the GitOps status reporting system.

Common Issue: Status shows OUT_OF_SYNC after deployment

Symptoms

• You pushed changes to the repo
• The deploy step succeeded
• But update-gitops-status shows OUT_OF_SYNC
• Changed files are not displayed

Root Causes

1. Deployment didn't actually succeed

The deploy step might have failed silently or the configuration wasn't applied correctly.

How to check:

• Look at the deploy step logs in Woodpecker
• SSH to the server and verify files match Git:

  diff /etc/rsyslog.conf files/rsyslog.conf
  diff -r /etc/rsyslog.d/ files/rsyslog.d/
  

2. Parsing issue with DRIFTED_FILES output

The script might not be correctly extracting the file list from Ansible's output.

How to debug: Run the status update script locally with debug mode:

export KEEP_PLAYBOOK_LOG=true
./update-gitops-status.sh

This will save the playbook output to drift-check-output.log. Check:

• Does the log contain DRIFTED_FILES= line?
• What does the line look like exactly?
• Are there ANSI color codes interfering?

Look for these debug lines in the output:

DEBUG: Searching for DRIFTED_FILES in playbook output...
DEBUG: Found DRIFTED_FILES pattern
DEBUG: Raw line: ...
DEBUG: Extracted value: '...'

3. Too many open files error

If you see "failed to create fsnotify watcher: too many open files":

Fixed in latest version:

• .woodpecker.yml now sets ANSIBLE_CALLBACKS_ENABLED="" and ANSIBLE_GATHERING=explicit
• update-gitops-status.sh uses ANSIBLE_CALLBACKS_ENABLED="" when running playbooks
• These settings prevent Ansible from exhausting inotify watches

If issue persists:

• The container might need privileged mode to adjust kernel parameters
• Or reduce Ansible parallelism in inventory settings

4. Ansible output format changed

If Ansible version changed, the debug output format might be different.

How to fix: Check drift-check-output.log and adjust the parsing in update-gitops-status.sh:

# Current parsing (line ~110 in update-gitops-status.sh):
DRIFTED_FILES_STR=$(echo "$DRIFTED_FILES_STR" | sed 's/.*DRIFTED_FILES=//' | sed 's/\x1b\[[0-9;]*m//g' | sed 's/".*$//' | xargs)

You might need to adjust the sed commands based on the actual format.

Testing the Fix

1. Test locally

# Set up SSH key
export SSH_PRIVATE_KEY="$(cat ~/.ssh/id_rsa)"

# Run the script with debug output
export KEEP_PLAYBOOK_LOG=true
./update-gitops-status.sh

# Check the log
cat drift-check-output.log | grep -A 2 "DRIFTED_FILES="

2. Test in Woodpecker

Push a small change and monitor the update-gitops-status step:

# Make a small comment change
echo "# Test change $(date)" >> files/rsyslog.conf

# Commit and push
git add files/rsyslog.conf
git commit -m "test: verify gitops status detection"
git push

# Watch the pipeline in Woodpecker UI
# The update-gitops-status step should:
# 1. Run deploy (apply.yml)
# 2. Run drift-check immediately after
# 3. Show SYNCED (because deploy just ran)
# 4. Show no drifted files

3. Test drift detection (manual change on server)

# SSH to the server
ssh rsyslog-lab

# Make a manual change
echo "# Manual change" >> /etc/rsyslog.conf

# Wait for the cron job to run (every 2 minutes)
# Or manually trigger it in Woodpecker

# The status should now show:
# - Status: OUT_OF_SYNC
# - Files: rsyslog.conf

Expected Behavior

After successful deployment (push to master)

Step 2/4: Analyzing drift detection results...
  ✓ Status: SYNCED - server configuration matches Git
  Total drift count: 0

Step 3/4: Building JSON payload...
  Generated JSON:
  {
    "repo": "rsyslog",
    "server": "rsyslog-lab",
    "sync_status": "SYNCED",
    "drift_count": 0,
    "files": [],
    "last_check": "2026-04-22T14:30:00Z"
  }

When drift is detected (cron job or manual server change)

Step 2/4: Analyzing drift detection results...
  ✗ Status: OUT OF SYNC - configuration drift detected
    - Drift detected in: rsyslog.conf
  Total drift count: 1

Step 3/4: Building JSON payload...
  Generated JSON:
  {
    "repo": "rsyslog",
    "server": "rsyslog-lab",
    "sync_status": "OUT_OF_SYNC",
    "drift_count": 1,
    "files": [
      {"name": "rsyslog.conf"}
    ],
    "last_check": "2026-04-22T14:32:00Z"
  }

Quick Reference

Enable debug mode

export KEEP_PLAYBOOK_LOG=true
./update-gitops-status.sh
cat drift-check-output.log

Manually run drift-check

ansible-playbook -i ansible/inventory/hosts.yml ansible/playbooks/drift-check.yml

Manually run deployment

ansible-playbook -i ansible/inventory/hosts.yml ansible/playbooks/apply.yml

Check server state

ssh rsyslog-lab "md5sum /etc/rsyslog.conf /etc/rsyslog.d/*.conf"
md5sum files/rsyslog.conf files/rsyslog.d/*.conf