mail-services/MAIL_STACK_README.md

# Lightweight Mail Stack for k3s GitOps Lab

Complete deployment of **Stalwart Mail Server** + **SnappyMail** webmail using GitOps with ArgoCD.

## 📋 Overview

This repository deploys a lightweight, modern mail stack designed for self-hosting and lab environments:

- **Stalwart Mail Server**: All-in-one mail server (SMTP, IMAP, Admin UI)
- **SnappyMail**: Modern, lightweight webmail client
- **GitOps**: Managed by ArgoCD
- **Storage**: NFS-based persistent volumes
- **Ingress**: Traefik for web UI access

## 🏗️ Architecture

```
┌─────────────────────────────────────────────────┐
│              Traefik Ingress                    │
│  mail.dvirlabs.com   webmail.dvirlabs.com      │
└───────────┬──────────────────────┬──────────────┘
            │                      │
            │                      │
    ┌───────▼────────┐    ┌────────▼────────┐
    │   Stalwart     │◄───│   SnappyMail    │
    │  Mail Server   │    │    Webmail      │
    │                │    │                 │
    │ • SMTP         │    │ Connects via:   │
    │ • IMAP         │    │ • IMAP:993      │
    │ • Admin UI     │    │ • SMTP:587      │
    └────────┬───────┘    └─────────────────┘
             │
             │
    ┌────────▼───────┐
    │  NFS Storage   │
    │  Mail Data     │
    └────────────────┘
```

## 📁 Repository Structure

```
mail-services/
├── argocd-apps/
│   ├── stalwart.yaml         # ArgoCD Application for Stalwart
│   └── snappymail.yaml       # ArgoCD Application for SnappyMail
├── charts/
│   ├── stalwart/             # Local Helm chart for Stalwart
│   │   ├── Chart.yaml
│   │   ├── values.yaml       # Default values
│   │   └── templates/
│   │       ├── namespace.yaml
│   │       ├── secret.yaml
│   │       ├── statefulset.yaml
│   │       ├── service.yaml
│   │       └── ingress.yaml
│   └── snappymail/           # Local Helm chart for SnappyMail
│       ├── Chart.yaml
│       ├── values.yaml       # Default values
│       └── templates/
│           ├── deployment.yaml
│           ├── pvc.yaml
│           ├── service.yaml
│           ├── ingress.yaml
│           └── configmap.yaml
└── manifests/
    ├── stalwart/
    │   └── values.yaml       # Custom values for dvirlabs.com
    └── snappymail/
        └── values.yaml       # Custom values for dvirlabs.com
```

## 🚀 Quick Start

### Prerequisites

- k3s cluster running
- ArgoCD installed and configured
- Traefik ingress controller
- NFS storage class (`nfs-client`)
- DNS records pointing to your cluster

### Step 1: Update Configuration

1. **Update ArgoCD Application manifests** with your Git repository URL:

```bash
# Edit both files and replace YOUR_USERNAME with your actual repo
vim argocd-apps/stalwart.yaml
vim argocd-apps/snappymail.yaml
```

2. **Change the Stalwart admin password**:

```bash
# Edit and set a strong password
vim manifests/stalwart/values.yaml
```

Find this section and change `CHANGE_ME_PLEASE_USE_STRONG_PASSWORD`:

```yaml
secret:
  create: true
  name: stalwart-credentials
  adminPassword: "YOUR_STRONG_PASSWORD_HERE"
```

3. **Update domain names** (if not using dvirlabs.com):

```bash
# Update in both files
vim manifests/stalwart/values.yaml
vim manifests/snappymail/values.yaml
```

### Step 2: Deploy with ArgoCD

```bash
# Apply ArgoCD Applications
kubectl apply -f argocd-apps/stalwart.yaml
kubectl apply -f argocd-apps/snappymail.yaml

# Check deployment status
kubectl get applications -n argocd

# Watch pods come up
kubectl get pods -n mail -w
```

### Step 3: Verify Deployment

```bash
# Check all resources in mail namespace
kubectl get all -n mail

# Check PVCs
kubectl get pvc -n mail

# Check ingresses
kubectl get ingress -n mail
```

Expected output:
```
NAME                       READY   STATUS    RESTARTS   AGE
pod/stalwart-0             1/1     Running   0          2m
pod/snappymail-xxx-xxx     1/1     Running   0          2m

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)
service/stalwart     ClusterIP   10.43.x.x       <none>        8080/TCP,25/TCP,587/TCP,993/TCP
service/snappymail   ClusterIP   10.43.x.x       <none>        8888/TCP

NAME                                    CLASS     HOSTS
ingress.networking.k8s.io/stalwart      traefik   mail.dvirlabs.com
ingress.networking.k8s.io/snappymail    traefik   webmail.dvirlabs.com
```

## 🌐 Access the Services

### Stalwart Admin UI

URL: `https://mail.dvirlabs.com`

Default credentials:
- Username: `admin@dvirlabs.com`
- Password: (the one you set in manifests/stalwart/values.yaml)

### SnappyMail Webmail

URL: `https://webmail.dvirlabs.com`

First-time setup:
1. Access the admin panel: `https://webmail.dvirlabs.com/?admin`
2. Default admin password: `12345` (change immediately!)
3. Configure mail server connection:
   - **IMAP Server**: `stalwart.mail.svc.cluster.local`
   - **IMAP Port**: `993`
   - **IMAP Security**: SSL/TLS
   - **SMTP Server**: `stalwart.mail.svc.cluster.local`
   - **SMTP Port**: `587`
   - **SMTP Security**: STARTTLS

## 📧 Configuring Real Mail Service

### Important: Cloudflare Tunnel Limitations

⚠️ **WARNING**: While Cloudflare Tunnel works fine for web UIs (admin panel and webmail), it **CANNOT** be used for actual email protocols (SMTP/IMAP).

**What works through Cloudflare Tunnel:**
- ✅ Stalwart admin UI (HTTPS)
- ✅ SnappyMail webmail (HTTPS)

**What does NOT work through Cloudflare Tunnel:**
- ❌ Receiving mail from other servers (SMTP port 25)
- ❌ Sending mail to other servers (SMTP port 25)
- ❌ External email clients (IMAP/SMTP)

### Required for Real Email

To receive and send real email, you need:

#### 1. DNS Records

```dns
; MX Record (Mail Exchange)
@           IN  MX  10  mail.dvirlabs.com.

; A Record (pointing to your public IP - NOT Cloudflare Tunnel)
mail        IN  A       YOUR_PUBLIC_IP

; SPF Record (Sender Policy Framework)
@           IN  TXT     "v=spf1 mx ~all"

; DMARC Record
_dmarc      IN  TXT     "v=DMARC1; p=quarantine; rua=mailto:admin@dvirlabs.com"

; DKIM Record (generated by Stalwart)
; Get this from Stalwart admin UI after setup
default._domainkey  IN  TXT  "v=DKIM1; k=rsa; p=YOUR_PUBLIC_KEY_HERE"
```

#### 2. Port Forwarding

You need to expose these ports directly (NOT through Cloudflare):

```
Port 25   (SMTP)  - Required for receiving mail from other servers
Port 587  (SMTP)  - Required for sending mail (submission)
Port 465  (SMTPS) - Optional, secure SMTP submission
Port 993  (IMAPS) - Required for IMAP access
Port 143  (IMAP)  - Optional, plaintext IMAP
```

**Option A: NodePort Service**

```bash
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: stalwart-external
  namespace: mail
spec:
  type: NodePort
  ports:
  - name: smtp
    port: 25
    targetPort: 25
    nodePort: 30025
  - name: submission
    port: 587
    targetPort: 587
    nodePort: 30587
  - name: imaps
    port: 993
    targetPort: 993
    nodePort: 30993
  selector:
    app.kubernetes.io/name: stalwart
EOF
```

Then forward ports 25, 587, 993 from your router to your k3s node on ports 30025, 30587, 30993.

**Option B: LoadBalancer with MetalLB**

If you have MetalLB configured:

```bash
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: stalwart-lb
  namespace: mail
spec:
  type: LoadBalancer
  loadBalancerIP: YOUR_LB_IP
  ports:
  - name: smtp
    port: 25
    targetPort: 25
  - name: submission
    port: 587
    targetPort: 587
  - name: imaps
    port: 993
    targetPort: 993
  selector:
    app.kubernetes.io/name: stalwart
EOF
```

#### 3. PTR (Reverse DNS) Record

Contact your ISP or VPS provider to set a PTR record:

```
YOUR_PUBLIC_IP  ->  mail.dvirlabs.com
```

This is **critical** for email deliverability. Without it, many servers will reject your mail.

## 🔧 Configuration Management

### Using External Secrets (Recommended for Production)

Instead of storing passwords in Git, use External Secrets Operator:

1. Install External Secrets Operator
2. Create a secret in your secret backend (Vault, AWS Secrets Manager, etc.)
3. Update manifests/stalwart/values.yaml:

```yaml
secret:
  create: false  # Don't create the secret
  name: stalwart-credentials  # Reference external secret
```

4. Create an ExternalSecret:

```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: stalwart-credentials
  namespace: mail
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: your-secret-store
    kind: SecretStore
  target:
    name: stalwart-credentials
  data:
  - secretKey: STALWART_ADMIN_PASSWORD
    remoteRef:
      key: mail/stalwart/admin-password
```

## 🛠️ Maintenance

### View Stalwart Logs

```bash
kubectl logs -n mail stalwart-0 -f
```

### View SnappyMail Logs

```bash
kubectl logs -n mail -l app.kubernetes.io/name=snappymail -f
```

### Access Stalwart Shell

```bash
kubectl exec -it -n mail stalwart-0 -- /bin/sh
```

### Backup Mail Data

```bash
# Backup Stalwart data
kubectl exec -n mail stalwart-0 -- tar czf /tmp/mail-backup.tar.gz /opt/stalwart-mail
kubectl cp mail/stalwart-0:/tmp/mail-backup.tar.gz ./mail-backup-$(date +%Y%m%d).tar.gz

# Backup SnappyMail config
kubectl exec -n mail -l app.kubernetes.io/name=snappymail -- tar czf /tmp/snappymail-backup.tar.gz /var/lib/snappymail
kubectl cp mail/snappymail-xxx:/tmp/snappymail-backup.tar.gz ./snappymail-backup-$(date +%Y%m%d).tar.gz
```

### Restore from Backup

```bash
# Restore Stalwart
kubectl cp ./mail-backup.tar.gz mail/stalwart-0:/tmp/mail-backup.tar.gz
kubectl exec -n mail stalwart-0 -- tar xzf /tmp/mail-backup.tar.gz -C /
kubectl rollout restart statefulset -n mail stalwart
```

## 🔍 Troubleshooting

### Pods Not Starting

```bash
# Check pod events
kubectl describe pod -n mail stalwart-0
kubectl describe pod -n mail -l app.kubernetes.io/name=snappymail

# Check PVC status
kubectl get pvc -n mail
```

### Ingress Not Working

```bash
# Check ingress status
kubectl describe ingress -n mail

# Check Traefik logs
kubectl logs -n kube-system -l app.kubernetes.io/name=traefik

# Test internal connectivity
kubectl run -it --rm debug --image=busybox -n mail -- wget -O- http://stalwart:8080
```

### SnappyMail Can't Connect to Stalwart

```bash
# Test IMAP connectivity from SnappyMail pod
kubectl exec -it -n mail -l app.kubernetes.io/name=snappymail -- nc -zv stalwart.mail.svc.cluster.local 993

# Check Stalwart service
kubectl get svc -n mail stalwart
```

### Email Not Being Delivered

Common issues:
1. **No PTR record**: Check reverse DNS
2. **Port 25 blocked**: Many ISPs block outbound port 25
3. **Missing SPF/DKIM/DMARC**: Check DNS records
4. **IP on blacklist**: Check https://mxtoolbox.com/blacklists.aspx

## 📊 Monitoring

### Check Mail Queue

```bash
# Access Stalwart admin UI
# https://mail.dvirlabs.com
# Navigate to Queue section
```

### Resource Usage

```bash
# Check pod resource usage
kubectl top pods -n mail

# Check PVC usage
kubectl exec -n mail stalwart-0 -- df -h /opt/stalwart-mail
```

## 🔐 Security Hardening

### Recommended Post-Deployment Steps

1. **Change SnappyMail admin password** immediately
2. **Enable fail2ban** for brute force protection
3. **Set up TLS certificates** with cert-manager (if not using Cloudflare)
4. **Enable DKIM signing** in Stalwart
5. **Configure rate limiting** in Stalwart
6. **Regular backups** of mail data
7. **Monitor logs** for suspicious activity

### TLS Certificates with cert-manager

If you want Let's Encrypt certificates instead of Cloudflare:

```yaml
# Add to ingress annotations
cert-manager.io/cluster-issuer: letsencrypt-prod
```

And configure TLS:

```yaml
tls:
  - hosts:
      - mail.dvirlabs.com
    secretName: stalwart-tls
```

## 📚 Additional Resources

- [Stalwart Documentation](https://stalw.art/docs)
- [SnappyMail Documentation](https://snappymail.eu/)
- [Email Deliverability Best Practices](https://www.mail-tester.com/)
- [DKIM Setup Guide](https://www.cloudflare.com/learning/dns/dns-records/dns-dkim-record/)

## ⚙️ Customization

### Adjust Storage Size

Edit `manifests/stalwart/values.yaml` or `manifests/snappymail/values.yaml`:

```yaml
persistence:
  size: 50Gi  # Increase as needed
```

### Change Resource Limits

Edit `manifests/*/values.yaml`:

```yaml
resources:
  requests:
    memory: "2Gi"
    cpu: "1000m"
  limits:
    memory: "8Gi"
    cpu: "4000m"
```

### Add Multiple Domains

Configure in Stalwart admin UI after deployment.

## 🤝 Contributing

This is a personal lab setup, but feel free to fork and adapt for your needs!

## 📝 License

MIT License - Use at your own risk

## ⚠️ Disclaimer

This setup is designed for lab/self-hosting environments. For production use:
- Use External Secrets for credentials
- Set up proper TLS certificates
- Configure backup automation
- Enable monitoring and alerting
- Review security best practices
- Test email deliverability thoroughly