Try to fix

WHATSAPP_DEBUGGING.md

@@ -0,0 +1,345 @@
+# WhatsApp Cloud API Debugging - Complete Implementation
+
+## Problem Summary
+
+WhatsApp Cloud API returns HTTP 200 with a wamid, but recipients don't receive messages.
+
+**Root Cause:** The initial API response (HTTP 200) only means Meta **accepted** the message, NOT that it was delivered. The actual delivery status comes from **webhook callbacks** which were not implemented.
+
+---
+
+## Solution Implemented
+
+### 1. ✅ Message Status Tracking
+
+**File:** `models.py`
+
+Added `WhatsAppMessage` model to track:
+- `wamid` - WhatsApp message ID from Meta
+- `status` - sent, delivered, read, failed
+- `event_id`, `guest_id` - Context for tracking
+- `error_code`, `error_title`, `error_message` - Failure details
+- Timestamps: `sent_at`, `delivered_at`, `read_at`, `failed_at`
+
+### 2. ✅ Webhook Endpoint
+
+**File:** `main.py`
+
+Added two webhook endpoints:
+
+```python
+GET  /whatsapp/webhook  # Webhook verification
+POST /whatsapp/webhook  # Status updates from Meta
+```
+
+**What it does:**
+- Receives delivery status updates from Meta
+- Updates message status in database
+- Logs all webhook payloads for debugging
+- Handles: sent, delivered, read, failed statuses
+- Captures error details when messages fail
+
+### 3. ✅ Enhanced WhatsApp Service
+
+**File:** `whatsapp.py`
+
+**Changes:**
+1. Added `event_id` and `guest_id` parameters to `send_by_template_key()`
+2. Saves messages to database after sending
+3. Supports dynamic image headers via `header_handle_key`
+4. Enhanced logging with full payload dumps
+
+### 4. ✅ Fixed hina_invitation Template
+
+**File:** `whatsapp_templates.py`
+
+**Configuration:**
+```python
+{
+    "meta_name": "hina_invitation",
+    "language_code": "he",
+    "header_type": "IMAGE",
+    "header_handle_key": "invitation_image_url",  # Dynamic image from params
+    "body_params": ["contact_name"],
+    "button_type": "URL",
+    "button_url": "https://invy.dvirlabs.com/guest/{{1}}",
+    "button_param_key": "event_id",
+}
+```
+
+### 5. ✅ Comprehensive Test Script
+
+**File:** `test_whatsapp_debug.py`
+
+**Features:**
+- Tests phone normalization
+- Validates template configuration
+- Sends test message with full logging
+- Shows webhook setup instructions
+- Provides clear debugging output
+
+### 6. ✅ Database Migration
+
+**File:** `add_whatsapp_messages_table.py`
+
+Creates the `whatsapp_messages` table.
+
+---
+
+## Setup Instructions
+
+### Step 1: Run Database Migration
+
+```bash
+cd backend
+python add_whatsapp_messages_table.py
+```
+
+### Step 2: Configure Webhook in Meta Business Manager
+
+1. Go to **Meta Business Manager** → **WhatsApp** → **Configuration** → **Webhook**
+
+2. **Callback URL:**
+   ```
+   https://invy.dvirlabs.com/whatsapp/webhook
+   ```
+
+3. **Verify Token:** 
+   - Set this in `.env` as `WHATSAPP_VERIFY_TOKEN=your_secret_token_here`
+   - Use the same value in Meta's webhook configuration
+
+4. Click **Verify and Save**
+
+5. **Subscribe to Fields:**
+   - ✅ messages
+
+### Step 3: Update Environment Variables
+
+Add to `.env`:
+```bash
+# Existing WhatsApp credentials
+WHATSAPP_ACCESS_TOKEN=your_meta_access_token
+WHATSAPP_PHONE_NUMBER_ID=your_phone_number_id
+WHATSAPP_API_VERSION=v20.0
+
+# NEW: Webhook verification token
+WHATSAPP_VERIFY_TOKEN=your_webhook_secret_token_here
+```
+
+### Step 4: Test the Integration
+
+```bash
+cd backend
+python test_whatsapp_debug.py
+```
+
+**What to check:**
+1. Phone normalization: ✓ All pass
+2. Template configuration: ✓ All correct
+3. Message send: ✓ Returns wamid
+4. Webhook configured: ✓ Verify token set
+
+### Step 5: Send Test Message
+
+The script will prompt you to send a test message to: `0504370045`
+
+**Expected Result:**
+```json
+{
+  "message_id": "wamid.HBgLOTcyNTA0MzcwMDQ1FQIAERgSQzI4QkE1RkQ5MzFFNTY1RTEwAA==",
+  "status": "sent",
+  "to": "+972504370045",
+  "template": "hina_invitation"
+}
+```
+
+---
+
+## How to Debug Delivery Issues
+
+### Check 1: Backend Logs
+
+Watch for webhook callbacks:
+```bash
+tail -f backend.log | grep "WhatsApp Webhook"
+```
+
+**Expected logs:**
+```
+[WhatsApp Webhook] Status update: wamid=xxx, status=delivered
+[WhatsApp Webhook] ✓ Updated message xxx to status: delivered
+```
+
+### Check 2: Database Status
+
+```sql
+SELECT 
+    wamid,
+    to_phone,
+    status,
+    template_name,
+    sent_at,
+    delivered_at,
+    failed_at,
+    error_code,
+    error_message
+FROM whatsapp_messages
+ORDER BY sent_at DESC
+LIMIT 10;
+```
+
+**Status meanings:**
+- `sent` - Meta accepted the message
+- `delivered` - Recipient's phone received it
+- `read` - Recipient opened the message
+- `failed` - Delivery failed (check error_code/error_message)
+
+### Check 3: Common Failure Reasons
+
+**If status stays "sent" and never reaches "delivered":**
+
+1. **Phone number not on WhatsApp**
+   - Error code: 131026
+   - Solution: Verify the recipient has WhatsApp
+
+2. **Template parameters mismatch**
+   - Error code: 132000
+   - Solution: Check template in Meta matches your params
+
+3. **Image URL not accessible**
+   - Error code: 132015
+   - Solution: Ensure image URL is publicly accessible
+
+4. **Phone number blocked your business**
+   - Error code: 131031
+   - Solution: Recipient must unblock your business number
+
+5. **Rate limiting**
+   - Error code: 130429
+   - Solution: Slow down sending rate
+
+### Check 4: Webhook Delivery Test
+
+Test webhook is receiving callbacks:
+```bash
+curl -X GET "https://invy.dvirlabs.com/whatsapp/webhook?hub.mode=subscribe&hub.verify_token=YOUR_TOKEN&hub.challenge=test123"
+```
+
+Expected: `test123` (echo challenge)
+
+---
+
+## Example: Sending hina_invitation
+
+```python
+from whatsapp import get_whatsapp_service
+
+service = get_whatsapp_service(db)
+
+params = {
+    "contact_name": "איילה חורב",
+    "event_id": "f3122a7d-1d7c-4cc1-955d-1c6b7358bd25",
+    "invitation_image_url": "https://api-invy.dvirlabs.com/uploads/1d32b5fbab0f494cae443b4188a83ca3.jpg",
+}
+
+result = await service.send_by_template_key(
+    template_key="hina_invitation",
+    to_phone="0504370045",
+    params=params,
+    event_id="f3122a7d-1d7c-4cc1-955d-1c6b7358bd25",
+    guest_id="guest-uuid-here",
+)
+
+print(f"Message ID: {result['message_id']}")
+```
+
+**Generated payload:**
+```json
+{
+  "messaging_product": "whatsapp",
+  "to": "+972504370045",
+  "type": "template",
+  "template": {
+    "name": "hina_invitation",
+    "language": { "code": "he" },
+    "components": [
+      {
+        "type": "header",
+        "parameters": [{
+          "type": "image",
+          "image": {
+            "link": "https://api-invy.dvirlabs.com/uploads/1d32b5fbab0f494cae443b4188a83ca3.jpg"
+          }
+        }]
+      },
+      {
+        "type": "body",
+        "parameters": [
+          { "type": "text", "text": "איילה חורב" }
+        ]
+      },
+      {
+        "type": "button",
+        "sub_type": "url",
+        "index": "0",
+        "parameters": [
+          { "type": "text", "text": "f3122a7d-1d7c-4cc1-955d-1c6b7358bd25" }
+        ]
+      }
+    ]
+  }
+}
+```
+
+---
+
+## Phone Normalization
+
+**Israeli numbers:**
+- `0504370045` → `+972504370045` ✓
+- `050-437-0045` → `+972504370045` ✓
+- `+972504370045` → `+972504370045` ✓
+- `972504370045` → `+972504370045` ✓
+
+---
+
+## Files Modified
+
+1. ✅ `models.py` - Added WhatsAppMessage model
+2. ✅ `main.py` - Added webhook endpoints, updated imports
+3. ✅ `whatsapp.py` - Added event/guest tracking, dynamic image support
+4. ✅ `whatsapp_templates.py` - Fixed hina_invitation template
+
+## Files Created
+
+1. ✅ `test_whatsapp_debug.py` - Comprehensive debugging script
+2. ✅ `add_whatsapp_messages_table.py` - Database migration
+3. ✅ `WHATSAPP_DEBUGGING.md` - This documentation
+
+---
+
+## Next Steps
+
+1. **Run migration:** `python add_whatsapp_messages_table.py`
+2. **Configure webhook in Meta Business Manager**
+3. **Set WHATSAPP_VERIFY_TOKEN in .env**
+4. **Run test script:** `python test_whatsapp_debug.py`
+5. **Monitor webhook logs for delivery status**
+6. **Check database for message status updates**
+
+---
+
+## Critical Understanding
+
+**HTTP 200 ≠ Message Delivered**
+
+When Meta returns HTTP 200, it means:
+- ✓ Request was valid
+- ✓ Message was queued for delivery
+- ✗ **NOT** that the recipient received it
+
+**Actual delivery confirmation comes from webhook callbacks.**
+
+Without webhooks, you'll never know if messages were delivered, read, or failed.
+
+**This is why webhook implementation is CRITICAL.**

backend/add_whatsapp_messages_table.py

@@ -0,0 +1,64 @@
+"""
+Database Migration: Add WhatsApp Message Tracking
+==================================================
+
+This migration adds the whatsapp_messages table for tracking
+message delivery status from Meta WhatsApp Cloud API webhooks.
+
+Run this migration:
+    python add_whatsapp_messages_table.py
+"""
+
+import sys
+from pathlib import Path
+
+# Add backend to path
+sys.path.insert(0, str(Path(__file__).parent))
+
+from database import engine, Base, SessionLocal
+from models import WhatsAppMessage
+import logging
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+
+def run_migration():
+    """Create the whatsapp_messages table"""
+    logger.info("Starting migration: Add whatsapp_messages table")
+    
+    try:
+        # Create only the WhatsAppMessage table
+        logger.info("Creating whatsapp_messages table...")
+        WhatsAppMessage.__table__.create(engine, checkfirst=True)
+        logger.info("✓ whatsapp_messages table created successfully")
+        
+        # Verify table exists
+        db = SessionLocal()
+        try:
+            result = db.execute("SELECT COUNT(*) FROM whatsapp_messages")
+            count = result.scalar()
+            logger.info(f"✓ Table verified: {count} messages in database")
+        finally:
+            db.close()
+        
+        logger.info("Migration completed successfully!")
+        
+    except Exception as e:
+        logger.error(f"✗ Migration failed: {e}")
+        raise
+
+
+if __name__ == "__main__":
+    print("="*80)
+    print("WhatsApp Messages Table Migration")
+    print("="*80)
+    print("\nThis will create the whatsapp_messages table for tracking")
+    print("delivery status of WhatsApp messages sent through Meta Cloud API.")
+    print()
+    
+    response = input("Proceed with migration? (yes/no): ").strip().lower()
+    if response in ['yes', 'y']:
+        run_migration()
+    else:
+        print("Migration cancelled.")

backend/test_whatsapp_debug.py

@@ -0,0 +1,313 @@
+"""
+WhatsApp Cloud API Debug Test Script
+=====================================
+
+This script tests WhatsApp message sending with full debugging to identify
+why messages return HTTP 200 but are not received by recipients.
+
+Usage:
+    python test_whatsapp_debug.py
+
+Target:
+    Phone: 0504370045 → 972504370045
+    Template: hina_invitation
+    Language: he
+"""
+
+import asyncio
+import sys
+import os
+import json
+import logging
+from pathlib import Path
+
+# Add backend to path
+sys.path.insert(0, str(Path(__file__).parent))
+
+# Configure detailed logging
+logging.basicConfig(
+    level=logging.DEBUG,
+    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
+)
+logger = logging.getLogger(__name__)
+
+# Load environment variables
+from dotenv import load_dotenv
+load_dotenv()
+
+from whatsapp import WhatsAppService, WhatsAppError
+from database import SessionLocal
+
+
+async def test_phone_normalization():
+    """Test that phone number normalization works correctly"""
+    print("\n" + "="*80)
+    print("STEP 1: Phone Number Normalization Test")
+    print("="*80)
+    
+    test_numbers = [
+        ("0504370045", "+972504370045"),
+        ("050-437-0045", "+972504370045"),
+        ("+972504370045", "+972504370045"),
+        ("972504370045", "+972504370045"),
+        ("0506118707", "+972506118707"),
+    ]
+    
+    for input_phone, expected in test_numbers:
+        normalized = WhatsAppService.normalize_phone_to_e164(input_phone)
+        status = "✓" if normalized == expected else "✗"
+        print(f"{status} {input_phone:20} → {normalized:20} (expected: {expected})")
+        if normalized != expected:
+            print(f"  ERROR: Normalization mismatch!")
+            return False
+    
+    print("\n✓ All phone normalizations correct")
+    return True
+
+
+async def test_template_configuration(db):
+    """Test that hina_invitation template is configured correctly"""
+    print("\n" + "="*80)
+    print("STEP 2: Template Configuration Test")
+    print("="*80)
+    
+    try:
+        from whatsapp_templates import get_template
+        
+        template = get_template(db, "hina_invitation")
+        
+        print("\nTemplate Configuration:")
+        print(f"  meta_name:        {template['meta_name']}")
+        print(f"  language_code:    {template['language_code']}")
+        print(f"  header_type:      {template.get('header_type', 'TEXT')}")
+        print(f"  header_params:    {template.get('header_params', [])}")
+        print(f"  header_handle_key:{template.get('header_handle_key', 'N/A')}")
+        print(f"  body_params:      {template.get('body_params', [])}")
+        print(f"  button_type:      {template.get('button_type', 'N/A')}")
+        print(f"  button_url:       {template.get('button_url', 'N/A')}")
+        print(f"  button_param_key: {template.get('button_param_key', 'N/A')}")
+        
+        # Validate template structure
+        issues = []
+        
+        if template.get('meta_name') != 'hina_invitation':
+            issues.append("meta_name must be 'hina_invitation'")
+        
+        if template.get('language_code') != 'he':
+            issues.append("language_code must be 'he'")
+        
+        if template.get('header_type') != 'IMAGE':
+            issues.append("header_type should be 'IMAGE' for this template")
+        
+        if not template.get('header_handle_key'):
+            issues.append("header_handle_key should be set for dynamic image")
+        
+        if template.get('body_params') != ['contact_name']:
+            issues.append("body_params should be ['contact_name']")
+        
+        if template.get('button_param_key') != 'event_id':
+            issues.append("button_param_key should be 'event_id'")
+        
+        if template.get('button_url') != 'https://invy.dvirlabs.com/guest/{{1}}':
+            issues.append("button_url should include dynamic parameter {{1}}")
+        
+        if issues:
+            print("\n✗ Template configuration issues found:")
+            for issue in issues:
+                print(f"  - {issue}")
+            return False
+        
+        print("\n✓ Template configuration is correct")
+        return True
+    
+    except Exception as e:
+        print(f"\n✗ Failed to load template: {e}")
+        return False
+
+
+async def test_send_message(db):
+    """Send a test WhatsApp message with full logging"""
+    print("\n" + "="*80)
+    print("STEP 3: Send Test WhatsApp Message")
+    print("="*80)
+    
+    # Test parameters
+    target_phone = "0504370045"
+    template_key = "hina_invitation"
+    
+    params = {
+        "contact_name": "בדיקה",
+        "event_id": "f3122a7d-1d7c-4cc1-955d-1c6b7358bd25",
+        "invitation_image_url": "https://api-invy.dvirlabs.com/uploads/1d32b5fbab0f494cae443b4188a83ca3.jpg",
+    }
+    
+    print(f"\nTest Parameters:")
+    print(f"  Phone:        {target_phone} → {WhatsAppService.normalize_phone_to_e164(target_phone)}")
+    print(f"  Template:     {template_key}")
+    print(f"  Language:     he")
+    print(f"  Contact Name: {params['contact_name']}")
+    print(f"  Event ID:     {params['event_id']}")
+    print(f"  Image URL:    {params['invitation_image_url']}")
+    
+    print(f"\nMeta API Configuration:")
+    print(f"  Access Token: {'***' + os.getenv('WHATSAPP_ACCESS_TOKEN', '')[-8:] if os.getenv('WHATSAPP_ACCESS_TOKEN') else 'NOT SET'}")
+    print(f"  Phone Number ID: {os.getenv('WHATSAPP_PHONE_NUMBER_ID', 'NOT SET')}")
+    print(f"  API Version:  {os.getenv('WHATSAPP_API_VERSION', 'v20.0')}")
+    
+    if not os.getenv('WHATSAPP_ACCESS_TOKEN') or not os.getenv('WHATSAPP_PHONE_NUMBER_ID'):
+        print("\n✗ ERROR: WhatsApp credentials not configured in .env file")
+        return False
+    
+    try:
+        service = WhatsAppService(db=db)
+        
+        print("\n" + "-"*80)
+        print("Sending message to Meta WhatsApp Cloud API...")
+        print("-"*80)
+        
+        result = await service.send_by_template_key(
+            template_key=template_key,
+            to_phone=target_phone,
+            params=params,
+            event_id=params['event_id'],
+            guest_id=None,
+        )
+        
+        print("\n" + "="*80)
+        print("✓ MESSAGE SENT SUCCESSFULLY!")
+        print("="*80)
+        print(f"\nResponse:")
+        print(json.dumps(result, indent=2, ensure_ascii=False))
+        
+        wamid = result.get('message_id', 'N/A')
+        print(f"\n✓ WhatsApp Message ID (wamid): {wamid}")
+        print(f"✓ Sent to: {result.get('to', 'N/A')}")
+        print(f"✓ Template: {result.get('template', 'N/A')}")
+        print(f"✓ Status: {result.get('status', 'N/A')}")
+        
+        print("\n" + "="*80)
+        print("IMPORTANT: Check WhatsApp Webhook for Delivery Status")
+        print("="*80)
+        print("""
+The message was accepted by Meta (HTTP 200), but this does NOT mean
+it was delivered. To confirm delivery:
+
+1. Configure webhook in Meta Business Manager:
+   URL: https://your-domain.com/whatsapp/webhook
+   Verify Token: (from WHATSAPP_VERIFY_TOKEN in .env)
+   
+2. Subscribe to 'messages' webhook events
+
+3. Watch the backend logs for webhook callbacks:
+   - Status: 'sent' = Meta accepted
+   - Status: 'delivered' = Recipient received
+   - Status: 'read' = Recipient opened
+   - Status: 'failed' = Delivery failed (check error details)
+
+4. Query the database for message status:
+   SELECT * FROM whatsapp_messages WHERE wamid = '{wamid}';
+
+If the message stays in 'sent' status and never reaches 'delivered',
+the problem is with Meta's delivery, not your code.
+
+Common reasons for non-delivery:
+- Phone number not registered with WhatsApp
+- Phone number blocked your business account
+- Template not approved or parameters mismatch
+- Image URL not accessible to Meta servers
+- Rate limiting or business account issues
+        """)
+        
+        return True
+    
+    except WhatsAppError as e:
+        print(f"\n✗ WhatsApp API Error: {e}")
+        return False
+    except Exception as e:
+        print(f"\n✗ Unexpected error: {e}")
+        logger.exception("Full traceback:")
+        return False
+
+
+async def check_webhook_configuration():
+    """Check if webhook is configured"""
+    print("\n" + "="*80)
+    print("STEP 4: Webhook Configuration Check")
+    print("="*80)
+    
+    verify_token = os.getenv('WHATSAPP_VERIFY_TOKEN', '')
+    
+    print(f"\nWebhook Settings:")
+    print(f"  Verify Token: {'***' + verify_token[-8:] if verify_token else 'NOT SET'}")
+    print(f"  Webhook URL:  https://your-domain.com/whatsapp/webhook")
+    
+    if not verify_token:
+        print("\n⚠ Warning: WHATSAPP_VERIFY_TOKEN not set in .env")
+        print("  You need this to verify the webhook with Meta")
+    else:
+        print("\n✓ Verify token is configured")
+    
+    print("\nWebhook Endpoints Available:")
+    print("  GET  /whatsapp/webhook - Webhook verification")
+    print("  POST /whatsapp/webhook - Status updates from Meta")
+    
+    print("\nTo configure webhook in Meta Business Manager:")
+    print("  1. Go to WhatsApp > Configuration > Webhook")
+    print("  2. Set Callback URL: https://your-domain.com/whatsapp/webhook")
+    print(f"  3. Set Verify Token: {verify_token if verify_token else '(SET THIS IN .env)'}")
+    print("  4. Click 'Verify and Save'")
+    print("  5. Subscribe to 'messages' webhook field")
+
+
+async def main():
+    """Run all debug tests"""
+    print("\n" + "="*80)
+    print("WhatsApp Cloud API Debugging Tool")
+    print("="*80)
+    print("""
+This script will:
+1. Test phone number normalization
+2. Validate template configuration  
+3. Send a test message with full logging
+4. Show webhook configuration status
+    """)
+    
+    db = SessionLocal()
+    try:
+        # Run tests
+        test1 = await test_phone_normalization()
+        test2 = await test_template_configuration(db)
+        
+        if not (test1 and test2):
+            print("\n✗ Pre-flight checks failed. Fix errors above before sending.")
+            return
+        
+        # Ask user to confirm send
+        print("\n" + "="*80)
+        response = input("Ready to send test message? (yes/no): ").strip().lower()
+        if response not in ['yes', 'y']:
+            print("Test cancelled.")
+            return
+        
+        test3 = await test_send_message(db)
+        await check_webhook_configuration()
+        
+        print("\n" + "="*80)
+        print("Debug Session Complete")
+        print("="*80)
+        
+        if test3:
+            print("\n✓ Message sent successfully")
+            print("\nNext steps:")
+            print("1. Check if recipient received the message on WhatsApp")
+            print("2. Watch backend logs for webhook status updates")
+            print("3. Query database: SELECT * FROM whatsapp_messages;")
+        else:
+            print("\n✗ Message send failed - see errors above")
+    
+    finally:
+        db.close()
+
+
+if __name__ == "__main__":
+    asyncio.run(main())