gitops-status-api/README.md

# GitOps Status API v2.0

**Generic multi-server status API for GitOps deployments**

A centralized status API that collects and exposes sync/drift status from multiple server types and groups. Designed to work with GitOps workflows and provide Grafana-friendly monitoring.

## Purpose

This API serves as a central status hub for GitOps-managed servers across your infrastructure:

- **Multi-server type support**: rsyslog, Splunk, IBM ITNM, nginx, or any future server type
- **Separation of concerns**: API only stores and exposes data - validation logic lives in each server's Ansible/config repo
- **Pipeline + Cron updates**: Separate endpoints for deployment status vs drift detection
- **Grafana integration**: Optimized output format for Grafana Infinity datasource
- **Backward compatible**: Existing rsyslog workflows continue to work

## Architecture

```
┌─────────────────────┐
│  Server Repo 1      │
│  (rsyslog)          │──┐
└─────────────────────┘  │
                         │
┌─────────────────────┐  │    ┌──────────────────┐       ┌─────────────┐
│  Server Repo 2      │  ├───►│  Status API      │◄──────│  Grafana    │
│  (Splunk)           │──┤    │  (This Project)  │       │  Dashboard  │
└─────────────────────┘  │    └──────────────────┘       └─────────────┘
                         │
┌─────────────────────┐  │
│  Server Repo 3      │  │
│  (nginx)            │──┘
└─────────────────────┘
```

Each server repository:
1. Runs validation in its own Ansible playbook/CI pipeline
2. Sends status updates to this API via POST /status/pipeline
3. Optionally runs cron drift checks that POST to /status/cron

Grafana queries this API via GET /status to visualize sync status across all servers.

## API Endpoints

### Pipeline Updates (Full Status)

**POST /status/pipeline**

Updates complete deployment status from CI/CD pipeline. Use this when deploying configuration changes.

**Required fields:**
- `server_group` - logical grouping (e.g., "rsyslog-prod")
- `host` - hostname (e.g., "rsyslog-01")
- `status` - one of: synced, out_of_sync, failed, unknown

**Optional fields:**
- `project_name` - project identifier
- `repo_name` - repository name
- `server_type` - type of server (rsyslog, splunk, nginx, etc.)
- `environment` - environment (prod, staging, dev, etc.)
- `changed_files` - array of changed file paths
- `validation_status` - validation result (passed, failed, skipped)
- `validation_message` - human-readable validation output
- `commit_sha` - git commit hash
- `branch` - git branch name
- `updated_by` - who/what triggered the update (e.g., "woodpecker", "jenkins")
- `last_pipeline_run` - ISO 8601 timestamp of pipeline execution

**Example:**

```bash
curl -X POST http://localhost:5000/status/pipeline \
  -H "Content-Type: application/json" \
  -d '{
    "project_name": "gitops-for-servers",
    "repo_name": "rsyslog",
    "server_group": "rsyslog-prod",
    "server_type": "rsyslog",
    "environment": "prod",
    "host": "rsyslog-01",
    "status": "synced",
    "changed_files": [],
    "validation_status": "passed",
    "validation_message": "rsyslog syntax validation passed",
    "commit_sha": "abc1234567890",
    "branch": "master",
    "updated_by": "woodpecker",
    "last_pipeline_run": "2026-04-26T10:30:00Z"
  }'
```

### Cron Updates (Drift Detection)

**POST /status/cron**

Updates only drift-related fields. Does **NOT** overwrite commit_sha, validation results, or other pipeline metadata.

**Required fields:**
- `server_group`
- `host`

**Optional fields:**
- `status` - current drift status
- `changed_files` - files that drifted
- `last_cron_check` - ISO 8601 timestamp

**Important:** The server must exist (created via /status/pipeline first).

**Example:**

```bash
curl -X POST http://localhost:5000/status/cron \
  -H "Content-Type: application/json" \
  -d '{
    "server_group": "rsyslog-prod",
    "host": "rsyslog-01",
    "status": "synced",
    "changed_files": [],
    "last_cron_check": "2026-04-26T11:00:00Z"
  }'
```

If drift detected:

```bash
curl -X POST http://localhost:5000/status/cron \
  -H "Content-Type: application/json" \
  -d '{
    "server_group": "rsyslog-prod",
    "host": "rsyslog-01",
    "status": "out_of_sync",
    "changed_files": ["/etc/rsyslog.conf", "/etc/rsyslog.d/50-default.conf"],
    "last_cron_check": "2026-04-26T11:00:00Z"
  }'
```

### Query Endpoints

**GET /status**

Returns all servers in Grafana-friendly flat array format.

```bash
curl http://localhost:5000/status
```

**Response:**

```json
[
  {
    "project_name": "gitops-for-servers",
    "repo_name": "rsyslog",
    "server_group": "rsyslog-prod",
    "server_type": "rsyslog",
    "environment": "prod",
    "host": "rsyslog-01",
    "status": "synced",
    "changed_files_count": 0,
    "changed_files": [],
    "validation_status": "passed",
    "validation_message": "rsyslog syntax validation passed",
    "last_pipeline_run": "2026-04-26T10:30:00Z",
    "last_cron_check": "2026-04-26T11:00:00Z",
    "commit_sha": "abc1234567890",
    "branch": "master",
    "updated_by": "woodpecker",
    "timestamp": "2026-04-26T11:00:00Z"
  },
  {
    "server_group": "splunk-prod",
    "server_type": "splunk",
    "host": "splunk-indexer-01",
    "status": "synced",
    ...
  }
]
```

**GET /status/{server_group}**

Returns all servers in a specific server group.

```bash
curl http://localhost:5000/status/rsyslog-prod
```

**GET /status/{server_group}/{host}**

Returns status for a specific server.

```bash
curl http://localhost:5000/status/rsyslog-prod/rsyslog-01
```

### Legacy Endpoints (Backward Compatibility)

**GET /api/status** and **POST /api/status**

Supports the old v1.x format for existing rsyslog workflows.

If you have existing pipelines using the old format, they will continue to work. The API automatically migrates old data to the new structure.

### Health Checks

**GET /health** - Liveness probe (always returns 200)

**GET /ready** - Readiness probe (checks file system)

## Grafana Infinity Configuration

### Setup

1. Install [Infinity datasource](https://grafana.com/grafana/plugins/yesoreyeram-infinity-datasource/) in Grafana
2. Add a new Infinity datasource pointing to your API:
   - URL: `http://gitops-status-api:5000`
   - Type: JSON

### Example Query

Create a new panel with these settings:

- **Type:** JSON
- **Source:** URL
- **URL:** `/status`
- **Method:** GET
- **Parser:** Backend
- **Root/Rows selector:** Leave empty (already flat array)

### Visualization Examples

**Table Panel:**
- Show all servers with status, last check times, validation results

**Stat Panel:**
- Filter by `status == "out_of_sync"` to show drift count
- Use threshold colors (green for 0, red for > 0)

**Bar Gauge:**
- Group by `server_group` and count `status` values

**Example Alert:**
```
Alert when changed_files_count > 0
```

## Adding a New Server Type

This API is generic - no code changes needed to support new server types.

### Steps:

1. **Create your server config repository** (e.g., `nginx-config`)

2. **Add validation in your Ansible playbook:**

```yaml
- name: Validate nginx config
  command: nginx -t
  register: validation_result
  
- name: Send status to API
  uri:
    url: "http://gitops-status-api:5000/status/pipeline"
    method: POST
    body_format: json
    body:
      project_name: "gitops-for-servers"
      repo_name: "nginx"
      server_group: "nginx-prod"
      server_type: "nginx"
      environment: "prod"
      host: "{{ inventory_hostname }}"
      status: "{{ 'synced' if validation_result.rc == 0 else 'failed' }}"
      validation_status: "{{ 'passed' if validation_result.rc == 0 else 'failed' }}"
      validation_message: "{{ validation_result.stderr }}"
      commit_sha: "{{ lookup('env', 'CI_COMMIT_SHA') }}"
      branch: "{{ lookup('env', 'CI_COMMIT_BRANCH') }}"
      updated_by: "woodpecker"
```

3. **Optional: Add drift detection cron job:**

```bash
#!/bin/bash
# Check if local config matches deployed config
if diff /etc/nginx/nginx.conf /opt/deployed/nginx.conf; then
  STATUS="synced"
  CHANGED=[]
else
  STATUS="out_of_sync"
  CHANGED='["/etc/nginx/nginx.conf"]'
fi

curl -X POST http://gitops-status-api:5000/status/cron \
  -H "Content-Type: application/json" \
  -d "{
    \"server_group\": \"nginx-prod\",
    \"host\": \"$(hostname)\",
    \"status\": \"$STATUS\",
    \"changed_files\": $CHANGED,
    \"last_cron_check\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"
  }"
```

That's it! No changes to the API needed.

## Working with Multiple Repositories

Each repository can manage different server types:

```
gitops-for-servers/
├── rsyslog/          # Repository for rsyslog servers
│   ├── playbook.yml  # Includes validation + API update
│   └── ...
├── splunk/           # Repository for Splunk servers
│   ├── playbook.yml
│   └── ...
├── nginx/            # Repository for nginx servers
│   ├── playbook.yml
│   └── ...
└── ibm-itnm/         # Repository for IBM ITNM
    ├── playbook.yml
    └── ...
```

Each repository independently:
1. Validates its own configuration (syntax, linting, etc.)
2. Posts status to the shared API
3. Uses consistent field names but server-specific values

The API automatically handles all server types without configuration.

## Storage Format

Data is stored in a JSON file (`/data/status.json` by default):

```json
{
  "version": "2.0",
  "servers": {
    "rsyslog-prod/rsyslog-01": {
      "project_name": "gitops-for-servers",
      "repo_name": "rsyslog",
      "server_group": "rsyslog-prod",
      "server_type": "rsyslog",
      "environment": "prod",
      "host": "rsyslog-01",
      "status": "synced",
      "changed_files": [],
      "validation_status": "passed",
      "validation_message": "rsyslog syntax validation passed",
      "commit_sha": "abc1234",
      "branch": "master",
      "updated_by": "woodpecker",
      "last_pipeline_run": "2026-04-26T10:30:00Z",
      "last_cron_check": "2026-04-26T11:00:00Z",
      "timestamp": "2026-04-26T11:00:00Z"
    },
    "rsyslog-prod/rsyslog-02": { ... },
    "splunk-prod/splunk-indexer-01": { ... }
  }
}
```

Updates are safe:
- New servers are added without affecting existing ones
- Pipeline updates overwrite deployment fields only
- Cron updates modify drift fields only
- Concurrent updates are handled by Flask's threading

## Local Development

```bash
# Install dependencies
pip install -r requirements.txt

# Run the app
python app.py

# Test pipeline update
curl -X POST http://localhost:5000/status/pipeline \
  -H "Content-Type: application/json" \
  -d '{"server_group":"test","host":"test-01","status":"synced"}'

# Test cron update
curl -X POST http://localhost:5000/status/cron \
  -H "Content-Type: application/json" \
  -d '{"server_group":"test","host":"test-01","status":"synced"}'

# Get all status
curl http://localhost:5000/status

# Get specific server
curl http://localhost:5000/status/test/test-01
```

## Docker Build & Deploy

```bash
# Build the image
docker build -t gitops-status-api:2.0.0 .

# Run locally with persistent storage
docker run -d \
  -p 5000:5000 \
  -v /var/lib/gitops-status:/data \
  --name gitops-status-api \
  gitops-status-api:2.0.0

# Test
curl http://localhost:5000/
```

## Push to Container Registry

```bash
# Login to your registry (Harbor, Docker Hub, etc.)
docker login harbor.your-domain.com

# Tag for registry
docker tag gitops-status-api:2.0.0 harbor.your-domain.com/gitops/status-api:2.0.0
docker tag gitops-status-api:2.0.0 harbor.your-domain.com/gitops/status-api:latest

# Push
docker push harbor.your-domain.com/gitops/status-api:2.0.0
docker push harbor.your-domain.com/gitops/status-api:latest
```

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `API_HOST` | `0.0.0.0` | Listen address |
| `API_PORT` | `5000` | Listen port |
| `STATUS_FILE` | `/data/status.json` | Path to status data file |
| `FLASK_ENV` | `production` | Flask environment |

## Kubernetes Deployment Example

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitops-status-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gitops-status-api
  template:
    metadata:
      labels:
        app: gitops-status-api
    spec:
      containers:
      - name: api
        image: harbor.your-domain.com/gitops/status-api:2.0.0
        ports:
        - containerPort: 5000
        volumeMounts:
        - name: data
          mountPath: /data
        livenessProbe:
          httpGet:
            path: /health
            port: 5000
          initialDelaySeconds: 10
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /ready
            port: 5000
          initialDelaySeconds: 5
          periodSeconds: 10
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: gitops-status-data
---
apiVersion: v1
kind: Service
metadata:
  name: gitops-status-api
spec:
  selector:
    app: gitops-status-api
  ports:
  - port: 5000
    targetPort: 5000
```

## Migration from v1.x

If you have an existing v1.x deployment:

1. **Automatic migration:** The API automatically detects and migrates old format on first load
2. **Legacy endpoints work:** Existing `/api/status` calls continue to function
3. **Update pipelines gradually:** Migrate to new `/status/pipeline` endpoint when ready

No data loss - old status.json is converted to new format automatically.

## Version

2.0.0