observability-stack/charts/gitops-status-server/README.md

# GitOps Status Server Helm Chart

A dual-container HTTP server that receives GitOps status updates via POST API and serves status information as JSON for monitoring and observability purposes.

## Overview

This chart deploys a two-container pod:
1. **Nginx** - Serves `/status.json` endpoint for monitoring tools and handles API routing
2. **Flask API** - Processes POST requests to `/api/status` and updates the status JSON

It's designed to be consumed by Grafana's Infinity datasource or other monitoring tools, and to receive updates from CI/CD pipelines like Woodpecker.

## Architecture

```
CI/CD Pipeline (Woodpecker)
    ↓
    POST /api/status
    ↓
Kubernetes Service (port 80)
    ↓
Nginx (port 8080)
    ├─→ /api/status → Proxies to Flask (localhost:5000)
    └─→ /status.json → Serves static file
    ↓
Shared Volume (emptyDir)
    ├─→ status.json (updated by Flask API)
    └─→ Read by Nginx
    ↓
Grafana Infinity Datasource
    Reads /status.json
```

## Features

- **API-driven updates**: POST endpoint for CI/CD pipelines to update status
- **Read-only serving**: Grafana-friendly JSON endpoint
- **Minimal footprint**: nginx-unprivileged + Python-Alpine with minimal resources
- **Secure by default**: Runs as non-root with restricted filesystems
- **Internal only**: ClusterIP service for cluster-internal access
- **ArgoCD compatible**: Init container auto-initializes status from ConfigMap
- **Production-ready**: Includes health checks, security contexts, and resource limits

## Installation

### Using Helm

```bash
# Install with default values
helm install gitops-status ./gitops-status-server

# Install with custom namespace
helm install gitops-status ./gitops-status-server -n observability-stack --create-namespace

# Install with custom values
helm install gitops-status ./gitops-status-server -f custom-values.yaml
```

### Using ArgoCD

Create an Application manifest:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: gitops-status-server
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/your-org/observability-stack
    targetRevision: main
    path: gitops-status-server
    helm:
      values: |
        replicaCount: 1
        statusJson:
          repo: "rsyslog"
          server: "rsyslog-lab"
          sync_status: "UNKNOWN"
```

## API Endpoints

### GET /status.json
Returns the current status JSON

```bash
curl http://gitops-status-server.observability-stack.svc.cluster.local:80/status.json
```

Response:
```json
{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "SYNCED",
  "drift_count": 0,
  "files": [],
  "last_check": "2026-04-21T10:30:00Z"
}
```

### POST /api/status
Updates the status with new data

```bash
curl -X POST http://gitops-status-server.observability-stack.svc.cluster.local:80/api/status \
  -H "Content-Type: application/json" \
  -d '{
    "repo": "rsyslog",
    "server": "rsyslog-lab",
    "sync_status": "OUT_OF_SYNC",
    "drift_count": 2,
    "files": [
      {"name": "rsyslog.conf"},
      {"name": "rsyslog.d/30-lab.conf"}
    ],
    "last_check": "2026-04-21T10:30:00Z"
  }'
```

Response (HTTP 200):
```json
{
  "success": true,
  "message": "Status updated successfully",
  "status": { ... }
}
```

### GET /health
Health check endpoint (returns HTTP 200)

```bash
curl http://gitops-status-server.observability-stack.svc.cluster.local:80/health
```

### GET /ready
Readiness check (verifies status file is readable)

```bash
curl http://gitops-status-server.observability-stack.svc.cluster.local:80/ready
```

## Integration with Woodpecker

The rsyslog CI/CD pipeline can update status by POSTing to the `/api/status` endpoint:

```bash
#!/bin/bash

GITOPS_STATUS_SERVER_URL="http://gitops-status-server.observability-stack.svc.cluster.local:80"

STATUS_JSON='{
  "repo": "rsyslog",
  "server": "rsyslog-lab",
  "sync_status": "SYNCED",
  "drift_count": 0,
  "files": [],
  "last_check": "2026-04-21T10:30:00Z"
}'

curl -X POST "$GITOPS_STATUS_SERVER_URL/api/status" \
  -H "Content-Type: application/json" \
  -d "$STATUS_JSON"
```

## Service Discovery

### Internal Kubernetes URL
```
http://gitops-status-server.observability-stack.svc.cluster.local:80/status.json
```

### Port Forwarding (for local testing)
```bash
kubectl port-forward -n observability-stack svc/gitops-status-server 8080:80
# Then access at http://localhost:8080/status.json
```

### NodePort (if service type is changed)
```bash
kubectl patch service -n observability-stack gitops-status-server -p '{"spec":{"type":"NodePort"}}'
# Then access at http://<node-ip>:<node-port>/status.json
```

## Configuration

See `values.yaml` for all configuration options:

- `replicaCount`: Number of replicas
- `image.repository`: Container image
- `image.tag`: Image tag
- `service.type`: Service type (ClusterIP, NodePort, LoadBalancer)
- `service.port`: Service port (default 80)
- `service.targetPort`: Container port (default 8080)
- `resources`: CPU/memory limits and requests
- `statusJson`: Default status JSON values
- `api.image.*`: Python/Flask image configuration

## Grafana Integration

### Infinity Datasource Configuration

1. Install Infinity datasource plugin:
   ```bash
   grafana-cli plugins install yesoreyeram-infinity-datasource
   ```

2. Add datasource with URL:
   ```
   http://gitops-status-server.observability-stack.svc.cluster.local:80/status.json
   ```

3. Create panels to visualize:
   - `sync_status`: Current synchronization state
   - `drift_count`: Number of drifted files
   - `files[]`: List of changed files
   - `last_check`: Timestamp of last check

### Example Query

```json
{
  "url": "http://gitops-status-server.observability-stack.svc.cluster.local:80/status.json",
  "format": "json"
}
```

## Security

- Runs as non-root user (UID 101)
- Read-only root filesystem (except for /tmp, /var/cache/nginx, /var/run)
- No privileged capabilities
- Network policies recommended for production
- Service Account with minimal RBAC

## Troubleshooting

### POST Request Returns 400 Error

**Issue**: "Invalid JSON" error

**Solution**: Verify JSON formatting with:
```bash
echo '{...}' | jq '.'
```

### POST Updates Not Appearing in GET Response

**Issue**: Update endpoint returns 200 but status.json isn't updated

**Possible causes**:
- Shared volume permission issue
- API container crashed after POST
- Status file permissions

**Debug**:
```bash
# Check logs
kubectl logs -f deployment/gitops-status-server -c api
kubectl logs -f deployment/gitops-status-server -c nginx

# Check shared volume
kubectl exec deployment/gitops-status-server -c nginx -- ls -la /usr/share/nginx/html/

# Test API directly (port-forward to 5000 first)
kubectl port-forward deployment/gitops-status-server 5000:5000
curl -X POST http://localhost:5000/api/status -H "Content-Type: application/json" -d '{...}'
```

### Connection Refused to gitops-status-server

**Issue**: Woodpecker can't reach the service

**Possible causes**:
- Service in different namespace
- Network policies blocking traffic
- Woodpecker outside cluster
- Service DNS name incorrect

**Solutions**:
- Verify service exists: `kubectl get svc gitops-status-server -n observability-stack`
- Use NodePort for external access (update service type in values)
- Use port-forward as a temporary solution
- Verify network policies allow traffic

## Performance

- **CPU**: 150m limit (100m nginx + 100m API)
- **Memory**: 192Mi limit (64Mi nginx + 128Mi API)
- **Startup time**: ~5 seconds (Flask app install + startup)
- **Update latency**: <100ms (direct file write)
- **Read performance**: <10ms (static file serving)

## License

Same as observability-stack repository
        statusJson:
          repo: "my-repo"
          server: "my-server"
          sync_status: "SYNCED"
          drift_count: 0
          files: []
          last_check: "2026-04-21T10:00:00Z"
  destination:
    server: https://kubernetes.default.svc
    namespace: monitoring
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
```

## Configuration

### Key Values

| Parameter | Description | Default |
|-----------|-------------|---------|
| `replicaCount` | Number of replicas | `1` |
| `image.repository` | Container image repository | `nginxinc/nginx-unprivileged` |
| `image.tag` | Container image tag | `1.25-alpine` |
| `service.type` | Kubernetes service type | `ClusterIP` |
| `service.port` | Service port | `80` |
| `service.targetPort` | Container target port | `8080` |
| `resources.limits.cpu` | CPU limit | `100m` |
| `resources.limits.memory` | Memory limit | `64Mi` |
| `statusJson` | JSON content to serve | See values.yaml |

### Custom Status JSON

Override the status JSON content in your values:

```yaml
statusJson:
  repo: "production-apps"
  server: "prod-cluster-01"
  sync_status: "SYNCED"
  drift_count: 2
  files:
    - "deployment.yaml"
    - "service.yaml"
  last_check: "2026-04-21T12:30:00Z"
```

## Usage

### Access the Status Endpoint

From inside the cluster:

```bash
# Using the service DNS name
curl http://gitops-status-server/status.json

# With namespace
curl http://gitops-status-server.monitoring.svc.cluster.local/status.json
```

### Grafana Infinity Datasource Configuration

1. Add an Infinity datasource in Grafana
2. Configure URL: `http://gitops-status-server.monitoring.svc.cluster.local/status.json`
3. Parser: JSON
4. Use fields from the JSON response in your dashboard

Example query fields:
- `sync_status` - Current sync status
- `drift_count` - Number of drifted resources
- `files` - List of changed files
- `last_check` - Timestamp of last check

## Updating Status Data

### Manual Update

Edit the ConfigMap directly:

```bash
kubectl edit configmap gitops-status-server -n monitoring
```

The deployment will automatically roll out with the new content due to the ConfigMap checksum annotation.

### Automated Update via Pipeline

Use `kubectl` in your CI/CD pipeline:

```bash
kubectl create configmap gitops-status-server \
  --from-file=status.json=./status.json \
  --dry-run=client -o yaml | kubectl apply -f -
```

### ArgoCD Hook (Advanced)

Create a PostSync hook that updates the ConfigMap with current sync status:

```yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: update-status
  annotations:
    argocd.argoproj.io/hook: PostSync
spec:
  template:
    spec:
      containers:
      - name: update
        image: bitnami/kubectl
        command:
        - /bin/sh
        - -c
        - |
          # Update status.json with current sync status
          kubectl patch configmap gitops-status-server \
            --patch '{"data":{"status.json":"..."}}'
      restartPolicy: Never
```

## Security Considerations

- Runs as non-root user (UID 101)
- Read-only root filesystem
- No privilege escalation
- Minimal capabilities (all dropped)
- No external network access required
- ClusterIP only (no external exposure)

## Resource Requirements

Minimal resource footprint suitable for small clusters:
- CPU: 50m request / 100m limit
- Memory: 32Mi request / 64Mi limit

## Troubleshooting

### Check pod status

```bash
kubectl get pods -l app.kubernetes.io/name=gitops-status-server
```

### View logs

```bash
kubectl logs -l app.kubernetes.io/name=gitops-status-server
```

### Test endpoint

```bash
kubectl run -it --rm curl --image=curlimages/curl --restart=Never -- \
  curl http://gitops-status-server/status.json
```

### Common Issues

**Pod not starting**: Check security context compatibility with your cluster's PSP/PSA policies.

**Empty response**: Verify the ConfigMap is mounted correctly:
```bash
kubectl describe pod -l app.kubernetes.io/name=gitops-status-server
```

**Service not accessible**: Ensure you're accessing from within the cluster and using the correct namespace.

## License

This chart is part of the observability-stack project.

## Maintainers

- DevOps Team