276 lines
7.0 KiB
Markdown
276 lines
7.0 KiB
Markdown
# Refactoring Summary
|
|
|
|
## Overview
|
|
|
|
Successfully refactored the GitOps Status API from a single-server, rsyslog-specific implementation to a generic, multi-server status API supporting any server type.
|
|
|
|
## Version Change
|
|
|
|
- **From:** v1.0.0 (single server, rsyslog-specific)
|
|
- **To:** v2.0.0 (multi-server, generic)
|
|
|
|
## Key Changes
|
|
|
|
### 1. Data Model
|
|
|
|
**Old (v1.x):**
|
|
```json
|
|
{
|
|
"repo": "rsyslog",
|
|
"server": "rsyslog-lab",
|
|
"sync_status": "SYNCED",
|
|
"drift_count": 0,
|
|
"deployed_files": [],
|
|
"drifted_files": [],
|
|
"last_check": "2026-04-26T00:00:00Z"
|
|
}
|
|
```
|
|
|
|
**New (v2.0):**
|
|
```json
|
|
{
|
|
"version": "2.0",
|
|
"servers": {
|
|
"rsyslog-prod/rsyslog-01": {
|
|
"project_name": "gitops-for-servers",
|
|
"repo_name": "rsyslog",
|
|
"server_group": "rsyslog-prod",
|
|
"server_type": "rsyslog",
|
|
"environment": "prod",
|
|
"host": "rsyslog-01",
|
|
"status": "synced",
|
|
"changed_files": [],
|
|
"validation_status": "passed",
|
|
"validation_message": "",
|
|
"commit_sha": "abc1234",
|
|
"branch": "master",
|
|
"updated_by": "woodpecker",
|
|
"last_pipeline_run": "2026-04-26T10:30:00Z",
|
|
"last_cron_check": "2026-04-26T11:00:00Z",
|
|
"timestamp": "2026-04-26T11:00:00Z"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### 2. New Endpoints
|
|
|
|
**Pipeline Updates (Full Status):**
|
|
```
|
|
POST /status/pipeline
|
|
```
|
|
Updates complete deployment status including validation, commit info, and changed files.
|
|
|
|
**Cron Updates (Drift Detection):**
|
|
```
|
|
POST /status/cron
|
|
```
|
|
Updates only drift-related fields without overwriting pipeline metadata.
|
|
|
|
**Query Endpoints:**
|
|
```
|
|
GET /status # All servers (Grafana-friendly)
|
|
GET /status/{server_group} # Servers in a group
|
|
GET /status/{server_group}/{host} # Specific server
|
|
```
|
|
|
|
**Legacy Endpoints (Backward Compatible):**
|
|
```
|
|
GET /api/status
|
|
POST /api/status
|
|
```
|
|
Automatically converts between old and new formats.
|
|
|
|
### 3. Separation of Concerns
|
|
|
|
**Pipeline updates** handle:
|
|
- Deployment status
|
|
- Validation results
|
|
- Commit SHA and branch
|
|
- Changed files from deployment
|
|
- Last pipeline run timestamp
|
|
|
|
**Cron updates** handle:
|
|
- Drift detection status
|
|
- Changed files from drift
|
|
- Last cron check timestamp
|
|
- Does NOT overwrite: commit_sha, validation_status, branch, etc.
|
|
|
|
### 4. Multi-Server Support
|
|
|
|
The API now supports unlimited servers of any type:
|
|
- rsyslog
|
|
- Splunk
|
|
- IBM ITNM
|
|
- nginx
|
|
- Any future server type
|
|
|
|
No code changes needed to add new server types - just send the appropriate payload.
|
|
|
|
### 5. Grafana Integration
|
|
|
|
The `GET /status` endpoint returns a flat array optimized for Grafana Infinity datasource:
|
|
|
|
```json
|
|
[
|
|
{
|
|
"server_group": "rsyslog-prod",
|
|
"server_type": "rsyslog",
|
|
"host": "rsyslog-01",
|
|
"status": "synced",
|
|
"changed_files_count": 0,
|
|
"validation_status": "passed",
|
|
...
|
|
}
|
|
]
|
|
```
|
|
|
|
### 6. Backward Compatibility
|
|
|
|
- Old `/api/status` endpoints still work
|
|
- Automatic migration from v1 to v2 format on first load
|
|
- Legacy payloads are converted to new structure automatically
|
|
- Existing rsyslog workflows continue without changes
|
|
|
|
## Testing Results
|
|
|
|
All tests passed successfully:
|
|
|
|
✅ **Health checks**: `/health` and `/ready` working
|
|
✅ **Pipeline updates**: Multiple server types (rsyslog, nginx) added successfully
|
|
✅ **Cron updates**: Drift detection without overwriting pipeline data
|
|
✅ **Query endpoints**: GET /status returns Grafana-friendly array
|
|
✅ **Group queries**: GET /status/{server_group} filters correctly
|
|
✅ **Host queries**: GET /status/{server_group}/{host} returns specific server
|
|
✅ **Legacy API**: Old format accepted and converted to new format
|
|
✅ **Data persistence**: Status stored in `/data/status.json` safely
|
|
|
|
## Example Usage
|
|
|
|
### Pipeline Update (rsyslog)
|
|
|
|
```bash
|
|
curl -X POST http://localhost:5000/status/pipeline \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"project_name": "gitops-for-servers",
|
|
"repo_name": "rsyslog",
|
|
"server_group": "rsyslog-prod",
|
|
"server_type": "rsyslog",
|
|
"environment": "prod",
|
|
"host": "rsyslog-01",
|
|
"status": "synced",
|
|
"validation_status": "passed",
|
|
"validation_message": "rsyslog syntax validation passed",
|
|
"commit_sha": "abc1234",
|
|
"branch": "master",
|
|
"updated_by": "woodpecker"
|
|
}'
|
|
```
|
|
|
|
### Cron Update (drift detected)
|
|
|
|
```bash
|
|
curl -X POST http://localhost:5000/status/cron \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"server_group": "rsyslog-prod",
|
|
"host": "rsyslog-01",
|
|
"status": "out_of_sync",
|
|
"changed_files": ["/etc/rsyslog.conf", "/etc/rsyslog.d/50-default.conf"]
|
|
}'
|
|
```
|
|
|
|
### Get All Status
|
|
|
|
```bash
|
|
curl http://localhost:5000/status
|
|
```
|
|
|
|
### Get Server Group
|
|
|
|
```bash
|
|
curl http://localhost:5000/status/rsyslog-prod
|
|
```
|
|
|
|
### Get Specific Server
|
|
|
|
```bash
|
|
curl http://localhost:5000/status/rsyslog-prod/rsyslog-01
|
|
```
|
|
|
|
## Files Modified
|
|
|
|
1. **app.py** - Complete refactor with new data model and endpoints
|
|
2. **README.md** - Comprehensive documentation update
|
|
3. **requirements.txt** - Updated Flask to v3.x for Python 3.14 compatibility
|
|
4. **test_api.sh** - New comprehensive test script (created)
|
|
5. **INTEGRATION_EXAMPLES.md** - Integration examples for Ansible, CI/CD, Cron (created)
|
|
|
|
## Migration Guide
|
|
|
|
### For Existing rsyslog Deployments
|
|
|
|
**Option 1: Continue using legacy endpoint**
|
|
No changes needed - keep using `/api/status`
|
|
|
|
**Option 2: Migrate to new endpoint**
|
|
Update your pipeline to use `/status/pipeline` with new field names:
|
|
- `sync_status` → `status` (lowercase: synced/out_of_sync/failed/unknown)
|
|
- `last_check` → `last_pipeline_run`
|
|
- Add new fields: `server_type`, `environment`, `validation_status`, etc.
|
|
|
|
### For New Server Types
|
|
|
|
1. Create server config repository
|
|
2. Add validation in Ansible playbook or CI pipeline
|
|
3. POST to `/status/pipeline` with appropriate fields
|
|
4. Optionally add cron drift detection using `/status/cron`
|
|
|
|
## Next Steps
|
|
|
|
1. **Test with current rsyslog flow**
|
|
- Run existing rsyslog pipeline
|
|
- Verify backward compatibility
|
|
- Gradually migrate to new endpoint
|
|
|
|
2. **Add other server types**
|
|
- Splunk
|
|
- IBM ITNM
|
|
- nginx
|
|
- Others as needed
|
|
|
|
3. **Set up Grafana dashboard**
|
|
- Install Infinity datasource
|
|
- Point to `GET /status`
|
|
- Create visualizations for sync status
|
|
|
|
4. **Deploy to production**
|
|
- Build Docker image
|
|
- Push to container registry
|
|
- Deploy to Kubernetes
|
|
- Configure persistent volume for `/data`
|
|
|
|
## Notes
|
|
|
|
- All datetime objects now use timezone-aware UTC format
|
|
- Fixed deprecation warnings for Python 3.14 compatibility
|
|
- Storage remains simple JSON file for easy debugging
|
|
- Thread-safe updates via Flask's threading
|
|
- Comprehensive error handling and validation
|
|
- API version displayed in root endpoint response
|
|
|
|
## Success Criteria Met
|
|
|
|
✅ Generic JSON model supporting any server type
|
|
✅ Separate pipeline and cron endpoints
|
|
✅ Backward compatible with existing rsyslog flow
|
|
✅ Grafana-friendly output format
|
|
✅ Simple, safe JSON storage
|
|
✅ Comprehensive documentation
|
|
✅ Example integrations for Ansible, CI/CD, Cron
|
|
✅ Clean, maintainable code
|
|
✅ Working test examples
|
|
|
|
The API is now production-ready and fully generic!
|