open-meteo-service/README.md

# Open-Meteo Coordinates Service

A FastAPI-based microservice that queries the Open-Meteo Geocoding API to retrieve coordinates for various cities. The service includes caching, Prometheus metrics, and Grafana dashboards for monitoring.

## Features

- **RESTful API** for retrieving city coordinates
- **Intelligent caching** to reduce external API calls
- **Prometheus metrics** for observability
- **Pre-configured Grafana dashboards** with 5 panels
- **Docker Compose** setup for easy deployment

## Prerequisites

- Docker
- Docker Compose

## Quick Start

1. **Clone the repository**
   ```bash
   cd open-meteo-service
   ```

2. **Start all services**
   ```bash
   docker compose up --build
   ```

3. **Access the services**
   - API: http://localhost:8000/docs
   - Prometheus: http://localhost:9090
   - Grafana: http://localhost:3000 (admin/admin)

## Helm (Kubernetes)

The Helm chart in the `helm` folder deploys the FastAPI app and, by default, Prometheus and Grafana.

### What it deploys
- **App**: FastAPI service with `/healthz` probes and Prometheus scrape annotations
- **Prometheus**: Scrapes the app metrics at `/metrics`
- **Grafana**: Pre-provisioned datasource and dashboard for the app metrics

### Prerequisites
- Kubernetes cluster
- Helm v3

### Install
```bash
helm install open-meteo-service ./helm
```

### Upgrade
```bash
helm upgrade open-meteo-service ./helm
```

### Access the services
```bash
kubectl port-forward svc/open-meteo-service-open-meteo-service 8080:8000
kubectl port-forward svc/open-meteo-service-open-meteo-service-prometheus 9090:9090
kubectl port-forward svc/open-meteo-service-open-meteo-service-grafana 3000:3000
```

### Values overview
You can customize the chart via `helm/values.yaml` or `--set` flags.

**App values**
- `image.repository`, `image.tag`, `image.pullPolicy`
- `service.type`, `service.port`
- `env.cacheFile`
- `persistence.enabled`, `persistence.size`, `persistence.storageClassName`

**Prometheus values**
- `prometheus.enabled`
- `prometheus.image`
- `prometheus.service.type`, `prometheus.service.port`
- `prometheus.persistence.enabled`, `prometheus.persistence.size`, `prometheus.persistence.storageClassName`

**Grafana values**
- `grafana.enabled`
- `grafana.image`
- `grafana.service.type`, `grafana.service.port`
- `grafana.adminUser`, `grafana.adminPassword`
- `grafana.persistence.enabled`, `grafana.persistence.size`, `grafana.persistence.storageClassName`

### Example: disable Grafana and Prometheus
```bash
helm install open-meteo-service ./helm \
  --set grafana.enabled=false \
  --set prometheus.enabled=false
```

### Example: enable persistence for all components
```bash
helm install open-meteo-service ./helm \
  --set persistence.enabled=true \
  --set prometheus.persistence.enabled=true \
  --set grafana.persistence.enabled=true
```

## GitLab CI Workflow

This project uses a GitLab CI/CD pipeline that automates building, versioning, deployment, and testing on an external k3s cluster (homelab).

### Pipeline Overview

The pipeline consists of two stages: **build** and **deploy**.

#### 1. Build Stage
- **Docker Image Build**: Builds the application Docker image using the project Dockerfile
- **Tagging Strategy**: Tags each image with a unique identifier combining branch and commit SHA:
  - Format: `<branch>-<shortsha>` (e.g., `main-a1b2c3d`, `feature-api-xyz1234`)
  - Uses `CI_COMMIT_REF_SLUG` (sanitized branch name) and `CI_COMMIT_SHORT_SHA`
- **Multi-Tag Push**: Pushes two tags to GitLab Container Registry:
  - `$IMAGE_REPO:<branch>-<shortsha>` - Specific version tag
  - `$IMAGE_REPO:latest` - Always points to the most recent build
- **Registry Authentication**: Uses GitLab's built-in CI variables (`CI_REGISTRY`, `CI_REGISTRY_USER`, `CI_REGISTRY_PASSWORD`)

#### 2. Deploy Stage
- **Values File Update**: 
  - Uses `yq` to update `open-meteo-service/values.yaml`
  - Sets `image.tag` to the newly built tag (`<branch>-<shortsha>`)
  - Keeps `image.repository` constant in the values file
  - Commits the change with message: `ci: bump image tag to <tag> [skip ci]`
  - Pushes back to the repository using `CI_JOB_TOKEN` for authentication
- **Helm Deployment**:
  - Deploys to namespace: `sandbox-gitlab`
  - Release name: `open-meteo-service-gitlab`
  - Uses the updated values.yaml file from the workspace
  - Waits for deployment to be ready (5-minute timeout)
- **Automated Tests**:
  - Port-forwards the service to localhost:8000
  - `/healthz` - Checks service health
  - `/coordinates` - Validates JSON response contains `"data"` field
  - `/metrics` - Ensures metrics endpoint is accessible
  - All tests must pass for the pipeline to succeed

### Preventing Infinite Loops

The pipeline includes safeguards to prevent infinite pipeline triggers:
- The deploy job includes a rule: `if: '$CI_COMMIT_MESSAGE =~ /^ci: bump image tag/'` with `when: never`
- The values bump commit includes `[skip ci]` to prevent triggering a new pipeline
- The deploy job uses the already-updated values.yaml in the current workspace

### Deployment Target

- **Cluster**: External k3s homelab cluster (not ephemeral)
- **Runner**: Self-hosted GitLab runner tagged with `homelab`
- **Namespace**: `sandbox-gitlab` (separate from ArgoCD/Harbor setups)
- **Ingress**: Configured for `open-meteo-gitlab.dvirlabs.com` (see values.yaml)

### Viewing Pipeline Status

1. Navigate to **CI/CD > Pipelines** in your GitLab project
2. Click on any pipeline run to see job details
3. Expand jobs to view real-time logs
4. The deploy job shows Helm output, rollout status, and test results

### Manual Deployment

To deploy manually with a specific tag:
```bash
# Option 1: Deploy using updated values.yaml
helm upgrade --install open-meteo-service-gitlab ./open-meteo-service \
  -n sandbox-gitlab --create-namespace

# Option 2: Override tag via --set
helm upgrade --install open-meteo-service-gitlab ./open-meteo-service \
  -n sandbox-gitlab --create-namespace \
  --set image.tag=main-a1b2c3d
```

## API Documentation

### Endpoints

#### `GET /coordinates`
Retrieve coordinates for all configured cities.

**Response:**
```json
{
  "source": "cache",
  "data": {
    "Tel Aviv": {
      "name": "Tel Aviv",
      "latitude": 32.08088,
      "longitude": 34.78057,
      "country": "Israel"
    },
    ...
  }
}
```

#### `GET /coordinates/{city}`
Retrieve coordinates for a specific city.

**Parameters:**
- `city` (path) - City name (e.g., "Ashkelon", "London")

**Example:**
```bash
curl http://localhost:8000/coordinates/Paris
```

**Response:**
```json
{
  "source": "open-meteo",
  "data": {
    "name": "Paris",
    "latitude": 48.85341,
    "longitude": 2.3488,
    "country": "France"
  }
}
```

#### `GET /metrics`
Prometheus metrics endpoint exposing service metrics.

**Example:**
```bash
curl http://localhost:8000/metrics
```

#### `GET /healthz`
Health check endpoint.

**Response:**
```json
{
  "status": "ok"
}
```

## Metrics

The service exposes the following Prometheus metrics:

### HTTP Metrics
- **`http_requests_total`** - Counter of total HTTP requests
  - Labels: `endpoint`, `method`, `status`
  
- **`http_request_duration_seconds`** - Histogram of request durations
  - Labels: `endpoint`, `method`

### Cache Metrics
- **`coordinates_cache_hits_total`** - Counter of cache hits
- **`coordinates_cache_misses_total`** - Counter of cache misses

### External API Metrics
- **`openmeteo_api_calls_total`** - Counter of calls to Open-Meteo Geocoding API
  - Labels: `city`

## Grafana Dashboard

The pre-configured dashboard includes 5 panels:

1. **Request Rate** - Requests per second by endpoint
2. **Request Duration p95** - 95th percentile latency
3. **Cache Hits vs Misses** - Cache effectiveness
4. **Open-Meteo Calls by City** - External API usage per city
5. **Requests by Status** - HTTP status code distribution

Access the dashboard at http://localhost:3000 after logging in with `admin/admin`.

## Caching

The service uses a local JSON file (`coordinates_cache.json`) to cache city coordinates:
- Reduces external API calls
- Shared across all API endpoints
- Persists between requests (not container restarts)
- Automatically updated when new cities are queried

## Development

### Project Structure
```
.
├── app/
│   ├── main.py          # FastAPI application
│   ├── service.py       # Business logic & caching
│   └── metrics.py       # Prometheus metrics definitions
├── grafana/
│   ├── provisioning/    # Auto-configured datasources & dashboards
│   └── dashboards/      # Dashboard JSON definitions
├── docker-compose.yml   # Service orchestration
├── Dockerfile           # Python app container
├── prometheus.yml       # Prometheus scrape configuration
└── requirements.txt     # Python dependencies
```

### Stop Services
```bash
docker compose down
```

### View Logs
```bash
docker compose logs -f open-meteo-service
```

### Rebuild After Code Changes
```bash
docker compose up --build
```

## Configuration

### Environment Variables
- `CACHE_FILE` - Path to cache file (default: `coordinates_cache.json`)

### Scrape Interval
Edit `prometheus.yml` to adjust the scrape interval (default: 15s).

## Testing

Generate test traffic to populate metrics:
```bash
# Test all endpoints
curl http://localhost:8000/coordinates
curl http://localhost:8000/coordinates/Paris
curl http://localhost:8000/coordinates/London

# Generate load
for i in {1..10}; do curl -s http://localhost:8000/coordinates > /dev/null; done
```

## License

MIT