# Implementation Plan ## Phase 1: Core Tracking Infrastructure - [x] 1. Set up database collections and indexes - Create MongoDB collections: `newsletter_sends`, `link_clicks`, `subscriber_activity` - Add indexes for performance: `tracking_id` (unique), `newsletter_id`, `subscriber_email`, `sent_at` - Write database initialization script - _Requirements: 1.3, 2.4_ - [x] 2. Implement tracking service - [x] 2.1 Create tracking ID generator - Write `generate_tracking_id()` function using UUID4 - Write `create_newsletter_tracking()` function to create tracking records - Add configuration for tracking enable/disable - _Requirements: 1.1, 2.1_ - [x] 2.2 Implement tracking pixel endpoint - Create Flask route `GET /api/track/pixel/` - Generate 1x1 transparent PNG (base64 encoded) - Log email open event to `newsletter_sends` collection - Handle multiple opens (update `last_opened_at` and `open_count`) - Return PNG with proper headers (Content-Type: image/png) - _Requirements: 1.2, 1.3, 1.4, 1.5_ - [x] 2.3 Implement link redirect endpoint - Create Flask route `GET /api/track/click/` - Look up original article URL from tracking record - Log click event to `link_clicks` collection - Redirect to original URL with 302 status - Handle invalid tracking_id (redirect to homepage) - Ensure redirect completes within 200ms - _Requirements: 2.2, 2.3, 2.4, 2.5_ - [x] 2.4 Write unit tests for tracking endpoints - Test pixel endpoint returns PNG for valid tracking_id - Test pixel endpoint returns PNG for invalid tracking_id (fail silently) - Test link redirect works correctly - Test link redirect handles invalid tracking_id - Test database logging for opens and clicks - _Requirements: 1.2, 1.4, 2.2, 2.3_ ## Phase 2: Newsletter Integration - [x] 3. Integrate tracking into sender service - [x] 3.1 Create tracking integration module - Write `inject_tracking_pixel()` function to add pixel to HTML - Write `replace_article_links()` function to replace links with tracking URLs - Write `generate_tracking_urls()` function to create tracking records for all links - Add tracking configuration to sender service - _Requirements: 1.1, 2.1_ - [x] 3.2 Modify newsletter sending flow - Update `send_newsletter()` to generate tracking IDs for each subscriber - Create tracking records in database before sending - Inject tracking pixel into newsletter HTML - Replace article links with tracking URLs - Store newsletter_id and tracking metadata - _Requirements: 1.1, 1.3, 2.1, 2.4_ - [x] 3.3 Update newsletter template - Ensure template supports dynamic tracking pixel injection - Ensure article links are properly structured for replacement - Add privacy notice to footer about tracking - _Requirements: 4.1_ - [x] 3.4 Test newsletter with tracking - Send test newsletter with tracking enabled - Verify tracking pixel is embedded correctly - Verify article links are replaced with tracking URLs - Test email open tracking works - Test link click tracking works - _Requirements: 1.1, 1.2, 2.1, 2.2_ ## Phase 3: Analytics and Reporting - [x] 4. Implement analytics service - [x] 4.1 Create analytics calculation functions - Write `get_open_rate(newsletter_id)` function - Write `get_click_rate(article_url)` function - Write `get_newsletter_metrics(newsletter_id)` function for overall stats - Write `get_article_performance(article_url)` function - _Requirements: 3.1, 3.2, 3.3, 3.4_ - [x] 4.2 Implement subscriber activity classification - Write `get_subscriber_activity_status(email)` function - Classify as 'active' (opened in last 30 days) - Classify as 'inactive' (no opens in 30-60 days) - Classify as 'dormant' (no opens in 60+ days) - Write `update_subscriber_activity_statuses()` batch function - _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5_ - [x] 4.3 Create analytics API endpoints - Create `GET /api/analytics/newsletter/` endpoint - Create `GET /api/analytics/article/` endpoint - Create `GET /api/analytics/subscriber/` endpoint - Create `POST /api/analytics/update-activity` endpoint - Return JSON with engagement metrics - _Requirements: 3.1, 3.2, 3.4, 5.4_ - [x] 4.4 Write unit tests for analytics - Test open rate calculation - Test click rate calculation - Test activity status classification - Test edge cases (no opens, no clicks) - _Requirements: 3.3, 3.4, 5.1, 5.2, 5.3_ ## Phase 4: Privacy and Compliance - [x] 5. Implement privacy features - [x] 5.1 Create data anonymization function - Write function to anonymize tracking data older than 90 days - Remove email addresses from old records - Keep aggregated metrics - Create scheduled task to run daily - _Requirements: 4.2_ - [x] 5.2 Implement user data deletion - Create `DELETE /api/tracking/subscriber/` endpoint - Delete all tracking records for subscriber - Delete from `newsletter_sends`, `link_clicks`, `subscriber_activity` - Return confirmation response - _Requirements: 4.3, 4.5_ - [x] 5.3 Add tracking opt-out support - Add `tracking_enabled` field to subscribers collection - Check opt-out status before creating tracking records - Skip tracking for opted-out subscribers - Update newsletter sending to respect opt-out - _Requirements: 4.4_ - [x] 5.4 Create anonymization endpoint - Create `POST /api/tracking/anonymize` endpoint - Trigger anonymization of old data - Return count of anonymized records - Add authentication/authorization - _Requirements: 4.2_ - [x] 5.5 Write privacy compliance tests - Test data anonymization works correctly - Test user data deletion removes all records - Test opt-out prevents tracking - Test anonymization preserves aggregated metrics - _Requirements: 4.2, 4.3, 4.4, 4.5_ ## Phase 5: Configuration and Documentation - [x] 6. Add configuration and environment setup - Add `TRACKING_ENABLED` to environment variables - Add `TRACKING_API_URL` configuration - Add `TRACKING_DATA_RETENTION_DAYS` configuration - Update `.env.template` with tracking variables - Update configuration documentation - _Requirements: All_ - [x] 7. Update database schema documentation - Document `newsletter_sends` collection schema - Document `link_clicks` collection schema - Document `subscriber_activity` collection schema - Add indexes documentation - Update DATABASE_SCHEMA.md - _Requirements: 1.3, 2.4, 5.4_ - [x] 8. Create tracking usage documentation - Document how to enable/disable tracking - Document analytics API endpoints - Document privacy features - Add examples of querying tracking data - Create README for tracking system - _Requirements: All_