A centralized status API that collects and exposes sync/drift status from multiple server types and groups. Designed to work with GitOps workflows and provide Grafana-friendly monitoring.

Purpose

This API serves as a central status hub for GitOps-managed servers across your infrastructure:

Multi-server type support: rsyslog, Splunk, IBM ITNM, nginx, or any future server type
Separation of concerns: API only stores and exposes data - validation logic lives in each server's Ansible/config repo
Pipeline + Cron updates: Separate endpoints for deployment status vs drift detection
Grafana integration: Optimized output format for Grafana Infinity datasource
Backward compatible: Existing rsyslog workflows continue to work

Architecture

┌─────────────────────┐
│  Server Repo 1      │
│  (rsyslog)          │──┐
└─────────────────────┘  │
                         │
┌─────────────────────┐  │    ┌──────────────────┐       ┌─────────────┐
│  Server Repo 2      │  ├───►│  Status API      │◄──────│  Grafana    │
│  (Splunk)           │──┤    │  (This Project)  │       │  Dashboard  │
└─────────────────────┘  │    └──────────────────┘       └─────────────┘
                         │
┌─────────────────────┐  │
│  Server Repo 3      │  │
│  (nginx)            │──┘
└─────────────────────┘

Each server repository:

Runs validation in its own Ansible playbook/CI pipeline
Sends status updates to this API via POST /status/pipeline
Optionally runs cron drift checks that POST to /status/cron

Grafana queries this API via GET /status to visualize sync status across all servers.

API Endpoints

Pipeline Updates (Full Status)

POST /status/pipeline

Updates complete deployment status from CI/CD pipeline. Use this when deploying configuration changes.

Required fields:

server_group - logical grouping (e.g., "rsyslog-prod")
host - hostname (e.g., "rsyslog-01")
status - one of: synced, out_of_sync, failed, unknown

Optional fields:

project_name - project identifier
repo_name - repository name
server_type - type of server (rsyslog, splunk, nginx, etc.)
environment - environment (prod, staging, dev, etc.)
changed_files - array of changed file paths
validation_status - validation result (passed, failed, skipped)
validation_message - human-readable validation output
commit_sha - git commit hash
branch - git branch name
updated_by - who/what triggered the update (e.g., "woodpecker", "jenkins")
last_pipeline_run - ISO 8601 timestamp of pipeline execution

Example:

curl -X POST http://localhost:5000/status/pipeline \
  -H "Content-Type: application/json" \
  -d '{
    "project_name": "gitops-for-servers",
    "repo_name": "rsyslog",
    "server_group": "rsyslog-prod",
    "server_type": "rsyslog",
    "environment": "prod",
    "host": "rsyslog-01",
    "status": "synced",
    "changed_files": [],
    "validation_status": "passed",
    "validation_message": "rsyslog syntax validation passed",
    "commit_sha": "abc1234567890",
    "branch": "master",
    "updated_by": "woodpecker",
    "last_pipeline_run": "2026-04-26T10:30:00Z"
  }'

Cron Updates (Drift Detection)

POST /status/cron

Updates only drift-related fields. Does NOT overwrite commit_sha, validation results, or other pipeline metadata.

Required fields:

server_group
host

Optional fields:

status - current drift status
changed_files - files that drifted
last_cron_check - ISO 8601 timestamp

Important: The server must exist (created via /status/pipeline first).

Example:

curl -X POST http://localhost:5000/status/cron \
  -H "Content-Type: application/json" \
  -d '{
    "server_group": "rsyslog-prod",
    "host": "rsyslog-01",
    "status": "synced",
    "changed_files": [],
    "last_cron_check": "2026-04-26T11:00:00Z"
  }'

If drift detected:

curl -X POST http://localhost:5000/status/cron \
  -H "Content-Type: application/json" \
  -d '{
    "server_group": "rsyslog-prod",
    "host": "rsyslog-01",
    "status": "out_of_sync",
    "changed_files": ["/etc/rsyslog.conf", "/etc/rsyslog.d/50-default.conf"],
    "last_cron_check": "2026-04-26T11:00:00Z"
  }'

Query Endpoints

GET /status

Returns all servers in Grafana-friendly flat array format.

curl http://localhost:5000/status

Response:

[
  {
    "project_name": "gitops-for-servers",
    "repo_name": "rsyslog",
    "server_group": "rsyslog-prod",
    "server_type": "rsyslog",
    "environment": "prod",
    "host": "rsyslog-01",
    "status": "synced",
    "changed_files_count": 0,
    "changed_files": [],
    "validation_status": "passed",
    "validation_message": "rsyslog syntax validation passed",
    "last_pipeline_run": "2026-04-26T10:30:00Z",
    "last_cron_check": "2026-04-26T11:00:00Z",
    "commit_sha": "abc1234567890",
    "branch": "master",
    "updated_by": "woodpecker",
    "timestamp": "2026-04-26T11:00:00Z"
  },
  {
    "server_group": "splunk-prod",
    "server_type": "splunk",
    "host": "splunk-indexer-01",
    "status": "synced",
    ...
  }
]

GET /status/{server_group}

Returns all servers in a specific server group.

curl http://localhost:5000/status/rsyslog-prod

GET /status/{server_group}/{host}

Returns status for a specific server.

curl http://localhost:5000/status/rsyslog-prod/rsyslog-01

Legacy Endpoints (Backward Compatibility)

GET /api/status and POST /api/status

Supports the old v1.x format for existing rsyslog workflows.

If you have existing pipelines using the old format, they will continue to work. The API automatically migrates old data to the new structure.

Health Checks

GET /health - Liveness probe (always returns 200)

GET /ready - Readiness probe (checks file system)

Grafana Infinity Configuration

Setup

Install Infinity datasource in Grafana
Add a new Infinity datasource pointing to your API:
- URL: http://gitops-status-api:5000
- Type: JSON

Example Query

Create a new panel with these settings:

Type: JSON
Source: URL
URL: /status
Method: GET
Parser: Backend
Root/Rows selector: Leave empty (already flat array)

Visualization Examples

Table Panel:

Show all servers with status, last check times, validation results

Stat Panel:

Filter by status == "out_of_sync" to show drift count
Use threshold colors (green for 0, red for > 0)

Bar Gauge:

Group by server_group and count status values

Example Alert:

Alert when changed_files_count > 0

Adding a New Server Type

This API is generic - no code changes needed to support new server types.

Steps:

Create your server config repository (e.g., nginx-config)
Add validation in your Ansible playbook:

- name: Validate nginx config
  command: nginx -t
  register: validation_result
  
- name: Send status to API
  uri:
    url: "http://gitops-status-api:5000/status/pipeline"
    method: POST
    body_format: json
    body:
      project_name: "gitops-for-servers"
      repo_name: "nginx"
      server_group: "nginx-prod"
      server_type: "nginx"
      environment: "prod"
      host: "{{ inventory_hostname }}"
      status: "{{ 'synced' if validation_result.rc == 0 else 'failed' }}"
      validation_status: "{{ 'passed' if validation_result.rc == 0 else 'failed' }}"
      validation_message: "{{ validation_result.stderr }}"
      commit_sha: "{{ lookup('env', 'CI_COMMIT_SHA') }}"
      branch: "{{ lookup('env', 'CI_COMMIT_BRANCH') }}"
      updated_by: "woodpecker"

Optional: Add drift detection cron job:

#!/bin/bash
# Check if local config matches deployed config
if diff /etc/nginx/nginx.conf /opt/deployed/nginx.conf; then
  STATUS="synced"
  CHANGED=[]
else
  STATUS="out_of_sync"
  CHANGED='["/etc/nginx/nginx.conf"]'
fi

curl -X POST http://gitops-status-api:5000/status/cron \
  -H "Content-Type: application/json" \
  -d "{
    \"server_group\": \"nginx-prod\",
    \"host\": \"$(hostname)\",
    \"status\": \"$STATUS\",
    \"changed_files\": $CHANGED,
    \"last_cron_check\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"
  }"

That's it! No changes to the API needed.

Working with Multiple Repositories

Each repository can manage different server types:

gitops-for-servers/
├── rsyslog/          # Repository for rsyslog servers
│   ├── playbook.yml  # Includes validation + API update
│   └── ...
├── splunk/           # Repository for Splunk servers
│   ├── playbook.yml
│   └── ...
├── nginx/            # Repository for nginx servers
│   ├── playbook.yml
│   └── ...
└── ibm-itnm/         # Repository for IBM ITNM
    ├── playbook.yml
    └── ...

Each repository independently:

Validates its own configuration (syntax, linting, etc.)
Posts status to the shared API
Uses consistent field names but server-specific values

The API automatically handles all server types without configuration.

Storage Format

Data is stored in a JSON file (/data/status.json by default):

{
  "version": "2.0",
  "servers": {
    "rsyslog-prod/rsyslog-01": {
      "project_name": "gitops-for-servers",
      "repo_name": "rsyslog",
      "server_group": "rsyslog-prod",
      "server_type": "rsyslog",
      "environment": "prod",
      "host": "rsyslog-01",
      "status": "synced",
      "changed_files": [],
      "validation_status": "passed",
      "validation_message": "rsyslog syntax validation passed",
      "commit_sha": "abc1234",
      "branch": "master",
      "updated_by": "woodpecker",
      "last_pipeline_run": "2026-04-26T10:30:00Z",
      "last_cron_check": "2026-04-26T11:00:00Z",
      "timestamp": "2026-04-26T11:00:00Z"
    },
    "rsyslog-prod/rsyslog-02": { ... },
    "splunk-prod/splunk-indexer-01": { ... }
  }
}

Updates are safe:

New servers are added without affecting existing ones
Pipeline updates overwrite deployment fields only
Cron updates modify drift fields only
Concurrent updates are handled by Flask's threading

Local Development

# Install dependencies
pip install -r requirements.txt

# Run the app
python app.py

# Test pipeline update
curl -X POST http://localhost:5000/status/pipeline \
  -H "Content-Type: application/json" \
  -d '{"server_group":"test","host":"test-01","status":"synced"}'

# Test cron update
curl -X POST http://localhost:5000/status/cron \
  -H "Content-Type: application/json" \
  -d '{"server_group":"test","host":"test-01","status":"synced"}'

# Get all status
curl http://localhost:5000/status

# Get specific server
curl http://localhost:5000/status/test/test-01

Docker Build & Deploy

# Build the image
docker build -t gitops-status-api:2.0.0 .

# Run locally with persistent storage
docker run -d \
  -p 5000:5000 \
  -v /var/lib/gitops-status:/data \
  --name gitops-status-api \
  gitops-status-api:2.0.0

# Test
curl http://localhost:5000/

Push to Container Registry

# Login to your registry (Harbor, Docker Hub, etc.)
docker login harbor.your-domain.com

# Tag for registry
docker tag gitops-status-api:2.0.0 harbor.your-domain.com/gitops/status-api:2.0.0
docker tag gitops-status-api:2.0.0 harbor.your-domain.com/gitops/status-api:latest

# Push
docker push harbor.your-domain.com/gitops/status-api:2.0.0
docker push harbor.your-domain.com/gitops/status-api:latest

Environment Variables

Variable	Default	Description
`API_HOST`	`0.0.0.0`	Listen address
`API_PORT`	`5000`	Listen port
`STATUS_FILE`	`/data/status.json`	Path to status data file
`FLASK_ENV`	`production`	Flask environment

Kubernetes Deployment Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitops-status-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gitops-status-api
  template:
    metadata:
      labels:
        app: gitops-status-api
    spec:
      containers:
      - name: api
        image: harbor.your-domain.com/gitops/status-api:2.0.0
        ports:
        - containerPort: 5000
        volumeMounts:
        - name: data
          mountPath: /data
        livenessProbe:
          httpGet:
            path: /health
            port: 5000
          initialDelaySeconds: 10
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /ready
            port: 5000
          initialDelaySeconds: 5
          periodSeconds: 10
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: gitops-status-data
---
apiVersion: v1
kind: Service
metadata:
  name: gitops-status-api
spec:
  selector:
    app: gitops-status-api
  ports:
  - port: 5000
    targetPort: 5000

Migration from v1.x

If you have an existing v1.x deployment:

Automatic migration: The API automatically detects and migrates old format on first load
Legacy endpoints work: Existing /api/status calls continue to function
Update pipelines gradually: Migrate to new /status/pipeline endpoint when ready

No data loss - old status.json is converted to new format automatically.

Version

2.0.0