gitops-status-api/README.md

GitOps Status API v2.0

Generic multi-server status API for GitOps deployments

A centralized status API that collects and exposes sync/drift status from multiple server types and groups. Designed to work with GitOps workflows and provide Grafana-friendly monitoring.

Purpose

This API serves as a central status hub for GitOps-managed servers across your infrastructure:

• Multi-server type support: rsyslog, Splunk, IBM ITNM, nginx, or any future server type
• Separation of concerns: API only stores and exposes data - validation logic lives in each server's Ansible/config repo
• Pipeline + Cron updates: Separate endpoints for deployment status vs drift detection
• Grafana integration: Optimized output format for Grafana Infinity datasource
• Backward compatible: Existing rsyslog workflows continue to work

Architecture

┌─────────────────────┐
│  Server Repo 1      │
│  (rsyslog)          │──┐
└─────────────────────┘  │
                         │
┌─────────────────────┐  │    ┌──────────────────┐       ┌─────────────┐
│  Server Repo 2      │  ├───►│  Status API      │◄──────│  Grafana    │
│  (Splunk)           │──┤    │  (This Project)  │       │  Dashboard  │
└─────────────────────┘  │    └──────────────────┘       └─────────────┘
                         │
┌─────────────────────┐  │
│  Server Repo 3      │  │
│  (nginx)            │──┘
└─────────────────────┘

Each server repository:

1. Runs validation in its own Ansible playbook/CI pipeline
2. Sends status updates to this API via POST /status/pipeline
3. Optionally runs cron drift checks that POST to /status/cron

Grafana queries this API via GET /status to visualize sync status across all servers.

API Endpoints

Pipeline Updates (Full Status)

POST /status/pipeline

Updates complete deployment status from CI/CD pipeline. Use this when deploying configuration changes.

Required fields:

• server_group - logical grouping (e.g., "rsyslog-prod")
• host - hostname (e.g., "rsyslog-01")
• status - one of: synced, out_of_sync, failed, unknown

Optional fields:

• project_name - project identifier
• repo_name - repository name
• server_type - type of server (rsyslog, splunk, nginx, etc.)
• environment - environment (prod, staging, dev, etc.)
• changed_files - array of changed file paths
• validation_status - validation result (passed, failed, skipped)
• validation_message - human-readable validation output
• commit_sha - git commit hash
• branch - git branch name
• updated_by - who/what triggered the update (e.g., "woodpecker", "jenkins")
• last_pipeline_run - ISO 8601 timestamp of pipeline execution

Example:

curl -X POST http://localhost:5000/status/pipeline \
  -H "Content-Type: application/json" \
  -d '{
    "project_name": "gitops-for-servers",
    "repo_name": "rsyslog",
    "server_group": "rsyslog-prod",
    "server_type": "rsyslog",
    "environment": "prod",
    "host": "rsyslog-01",
    "status": "synced",
    "changed_files": [],
    "validation_status": "passed",
    "validation_message": "rsyslog syntax validation passed",
    "commit_sha": "abc1234567890",
    "branch": "master",
    "updated_by": "woodpecker",
    "last_pipeline_run": "2026-04-26T10:30:00Z"
  }'

Cron Updates (Drift Detection)

POST /status/cron

Updates only drift-related fields. Does NOT overwrite commit_sha, validation results, or other pipeline metadata.

Required fields:

• server_group
• host

Optional fields:

• status - current drift status
• changed_files - files that drifted
• last_cron_check - ISO 8601 timestamp

Important: The server must exist (created via /status/pipeline first).

Example:

curl -X POST http://localhost:5000/status/cron \
  -H "Content-Type: application/json" \
  -d '{
    "server_group": "rsyslog-prod",
    "host": "rsyslog-01",
    "status": "synced",
    "changed_files": [],
    "last_cron_check": "2026-04-26T11:00:00Z"
  }'

If drift detected:

curl -X POST http://localhost:5000/status/cron \
  -H "Content-Type: application/json" \
  -d '{
    "server_group": "rsyslog-prod",
    "host": "rsyslog-01",
    "status": "out_of_sync",
    "changed_files": ["/etc/rsyslog.conf", "/etc/rsyslog.d/50-default.conf"],
    "last_cron_check": "2026-04-26T11:00:00Z"
  }'

Query Endpoints

GET /status

Returns all servers in Grafana-friendly flat array format.

curl http://localhost:5000/status

Response:

[
  {
    "project_name": "gitops-for-servers",
    "repo_name": "rsyslog",
    "server_group": "rsyslog-prod",
    "server_type": "rsyslog",
    "environment": "prod",
    "host": "rsyslog-01",
    "status": "synced",
    "changed_files_count": 0,
    "changed_files": [],
    "validation_status": "passed",
    "validation_message": "rsyslog syntax validation passed",
    "last_pipeline_run": "2026-04-26T10:30:00Z",
    "last_cron_check": "2026-04-26T11:00:00Z",
    "commit_sha": "abc1234567890",
    "branch": "master",
    "updated_by": "woodpecker",
    "timestamp": "2026-04-26T11:00:00Z"
  },
  {
    "server_group": "splunk-prod",
    "server_type": "splunk",
    "host": "splunk-indexer-01",
    "status": "synced",
    ...
  }
]

GET /status/{server_group}

Returns all servers in a specific server group.

curl http://localhost:5000/status/rsyslog-prod

GET /status/{server_group}/{host}

Returns status for a specific server.

curl http://localhost:5000/status/rsyslog-prod/rsyslog-01

Legacy Endpoints (Backward Compatibility)

GET /api/status and POST /api/status

Supports the old v1.x format for existing rsyslog workflows.

If you have existing pipelines using the old format, they will continue to work. The API automatically migrates old data to the new structure.

Health Checks

GET /health - Liveness probe (always returns 200)

GET /ready - Readiness probe (checks file system)

Grafana Infinity Configuration

Setup

1. Install Infinity datasource in Grafana
2. Add a new Infinity datasource pointing to your API:
   • URL: http://gitops-status-api:5000
   • Type: JSON

Example Query

Create a new panel with these settings:

• Type: JSON
• Source: URL
• URL: /status
• Method: GET
• Parser: Backend
• Root/Rows selector: Leave empty (already flat array)

Visualization Examples

Table Panel:

• Show all servers with status, last check times, validation results

Stat Panel:

• Filter by status == "out_of_sync" to show drift count
• Use threshold colors (green for 0, red for > 0)

Bar Gauge:

• Group by server_group and count status values

Example Alert:

Alert when changed_files_count > 0

Adding a New Server Type

This API is generic - no code changes needed to support new server types.

Steps:

1. Create your server config repository (e.g., nginx-config)

2. Add validation in your Ansible playbook:

- name: Validate nginx config
  command: nginx -t
  register: validation_result
  
- name: Send status to API
  uri:
    url: "http://gitops-status-api:5000/status/pipeline"
    method: POST
    body_format: json
    body:
      project_name: "gitops-for-servers"
      repo_name: "nginx"
      server_group: "nginx-prod"
      server_type: "nginx"
      environment: "prod"
      host: "{{ inventory_hostname }}"
      status: "{{ 'synced' if validation_result.rc == 0 else 'failed' }}"
      validation_status: "{{ 'passed' if validation_result.rc == 0 else 'failed' }}"
      validation_message: "{{ validation_result.stderr }}"
      commit_sha: "{{ lookup('env', 'CI_COMMIT_SHA') }}"
      branch: "{{ lookup('env', 'CI_COMMIT_BRANCH') }}"
      updated_by: "woodpecker"

3. Optional: Add drift detection cron job:

#!/bin/bash
# Check if local config matches deployed config
if diff /etc/nginx/nginx.conf /opt/deployed/nginx.conf; then
  STATUS="synced"
  CHANGED=[]
else
  STATUS="out_of_sync"
  CHANGED='["/etc/nginx/nginx.conf"]'
fi

curl -X POST http://gitops-status-api:5000/status/cron \
  -H "Content-Type: application/json" \
  -d "{
    \"server_group\": \"nginx-prod\",
    \"host\": \"$(hostname)\",
    \"status\": \"$STATUS\",
    \"changed_files\": $CHANGED,
    \"last_cron_check\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"
  }"

That's it! No changes to the API needed.

Working with Multiple Repositories

Each repository can manage different server types:

gitops-for-servers/
├── rsyslog/          # Repository for rsyslog servers
│   ├── playbook.yml  # Includes validation + API update
│   └── ...
├── splunk/           # Repository for Splunk servers
│   ├── playbook.yml
│   └── ...
├── nginx/            # Repository for nginx servers
│   ├── playbook.yml
│   └── ...
└── ibm-itnm/         # Repository for IBM ITNM
    ├── playbook.yml
    └── ...

Each repository independently:

1. Validates its own configuration (syntax, linting, etc.)
2. Posts status to the shared API
3. Uses consistent field names but server-specific values

The API automatically handles all server types without configuration.

Storage Format

Data is stored in a JSON file (/data/status.json by default):

{
  "version": "2.0",
  "servers": {
    "rsyslog-prod/rsyslog-01": {
      "project_name": "gitops-for-servers",
      "repo_name": "rsyslog",
      "server_group": "rsyslog-prod",
      "server_type": "rsyslog",
      "environment": "prod",
      "host": "rsyslog-01",
      "status": "synced",
      "changed_files": [],
      "validation_status": "passed",
      "validation_message": "rsyslog syntax validation passed",
      "commit_sha": "abc1234",
      "branch": "master",
      "updated_by": "woodpecker",
      "last_pipeline_run": "2026-04-26T10:30:00Z",
      "last_cron_check": "2026-04-26T11:00:00Z",
      "timestamp": "2026-04-26T11:00:00Z"
    },
    "rsyslog-prod/rsyslog-02": { ... },
    "splunk-prod/splunk-indexer-01": { ... }
  }
}

Updates are safe:

• New servers are added without affecting existing ones
• Pipeline updates overwrite deployment fields only
• Cron updates modify drift fields only
• Concurrent updates are handled by Flask's threading

Local Development

# Install dependencies
pip install -r requirements.txt

# Run the app
python app.py

# Test pipeline update
curl -X POST http://localhost:5000/status/pipeline \
  -H "Content-Type: application/json" \
  -d '{"server_group":"test","host":"test-01","status":"synced"}'

# Test cron update
curl -X POST http://localhost:5000/status/cron \
  -H "Content-Type: application/json" \
  -d '{"server_group":"test","host":"test-01","status":"synced"}'

# Get all status
curl http://localhost:5000/status

# Get specific server
curl http://localhost:5000/status/test/test-01

Docker Build & Deploy

# Build the image
docker build -t gitops-status-api:2.0.0 .

# Run locally with persistent storage
docker run -d \
  -p 5000:5000 \
  -v /var/lib/gitops-status:/data \
  --name gitops-status-api \
  gitops-status-api:2.0.0

# Test
curl http://localhost:5000/

Push to Container Registry

# Login to your registry (Harbor, Docker Hub, etc.)
docker login harbor.your-domain.com

# Tag for registry
docker tag gitops-status-api:2.0.0 harbor.your-domain.com/gitops/status-api:2.0.0
docker tag gitops-status-api:2.0.0 harbor.your-domain.com/gitops/status-api:latest

# Push
docker push harbor.your-domain.com/gitops/status-api:2.0.0
docker push harbor.your-domain.com/gitops/status-api:latest

Environment Variables

Variable	Default	Description
API_HOST	0.0.0.0	Listen address
API_PORT	5000	Listen port
STATUS_FILE	/data/status.json	Path to status data file
FLASK_ENV	production	Flask environment

Kubernetes Deployment Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitops-status-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gitops-status-api
  template:
    metadata:
      labels:
        app: gitops-status-api
    spec:
      containers:
      - name: api
        image: harbor.your-domain.com/gitops/status-api:2.0.0
        ports:
        - containerPort: 5000
        volumeMounts:
        - name: data
          mountPath: /data
        livenessProbe:
          httpGet:
            path: /health
            port: 5000
          initialDelaySeconds: 10
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /ready
            port: 5000
          initialDelaySeconds: 5
          periodSeconds: 10
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: gitops-status-data
---
apiVersion: v1
kind: Service
metadata:
  name: gitops-status-api
spec:
  selector:
    app: gitops-status-api
  ports:
  - port: 5000
    targetPort: 5000

Migration from v1.x

If you have an existing v1.x deployment:

1. Automatic migration: The API automatically detects and migrates old format on first load
2. Legacy endpoints work: Existing /api/status calls continue to function
3. Update pipelines gradually: Migrate to new /status/pipeline endpoint when ready

No data loss - old status.json is converted to new format automatically.

Version

2.0.0