Munich-news/.kiro/specs/email-tracking/tasks.md

# Implementation Plan

## Phase 1: Core Tracking Infrastructure

- [x] 1. Set up database collections and indexes
  - Create MongoDB collections: `newsletter_sends`, `link_clicks`, `subscriber_activity`
  - Add indexes for performance: `tracking_id` (unique), `newsletter_id`, `subscriber_email`, `sent_at`
  - Write database initialization script
  - _Requirements: 1.3, 2.4_

- [x] 2. Implement tracking service
- [x] 2.1 Create tracking ID generator
  - Write `generate_tracking_id()` function using UUID4
  - Write `create_newsletter_tracking()` function to create tracking records
  - Add configuration for tracking enable/disable
  - _Requirements: 1.1, 2.1_

- [x] 2.2 Implement tracking pixel endpoint
  - Create Flask route `GET /api/track/pixel/<tracking_id>`
  - Generate 1x1 transparent PNG (base64 encoded)
  - Log email open event to `newsletter_sends` collection
  - Handle multiple opens (update `last_opened_at` and `open_count`)
  - Return PNG with proper headers (Content-Type: image/png)
  - _Requirements: 1.2, 1.3, 1.4, 1.5_

- [x] 2.3 Implement link redirect endpoint
  - Create Flask route `GET /api/track/click/<tracking_id>`
  - Look up original article URL from tracking record
  - Log click event to `link_clicks` collection
  - Redirect to original URL with 302 status
  - Handle invalid tracking_id (redirect to homepage)
  - Ensure redirect completes within 200ms
  - _Requirements: 2.2, 2.3, 2.4, 2.5_

- [x] 2.4 Write unit tests for tracking endpoints
  - Test pixel endpoint returns PNG for valid tracking_id
  - Test pixel endpoint returns PNG for invalid tracking_id (fail silently)
  - Test link redirect works correctly
  - Test link redirect handles invalid tracking_id
  - Test database logging for opens and clicks
  - _Requirements: 1.2, 1.4, 2.2, 2.3_

## Phase 2: Newsletter Integration

- [x] 3. Integrate tracking into sender service
- [x] 3.1 Create tracking integration module
  - Write `inject_tracking_pixel()` function to add pixel to HTML
  - Write `replace_article_links()` function to replace links with tracking URLs
  - Write `generate_tracking_urls()` function to create tracking records for all links
  - Add tracking configuration to sender service
  - _Requirements: 1.1, 2.1_

- [x] 3.2 Modify newsletter sending flow
  - Update `send_newsletter()` to generate tracking IDs for each subscriber
  - Create tracking records in database before sending
  - Inject tracking pixel into newsletter HTML
  - Replace article links with tracking URLs
  - Store newsletter_id and tracking metadata
  - _Requirements: 1.1, 1.3, 2.1, 2.4_

- [x] 3.3 Update newsletter template
  - Ensure template supports dynamic tracking pixel injection
  - Ensure article links are properly structured for replacement
  - Add privacy notice to footer about tracking
  - _Requirements: 4.1_

- [x] 3.4 Test newsletter with tracking
  - Send test newsletter with tracking enabled
  - Verify tracking pixel is embedded correctly
  - Verify article links are replaced with tracking URLs
  - Test email open tracking works
  - Test link click tracking works
  - _Requirements: 1.1, 1.2, 2.1, 2.2_

## Phase 3: Analytics and Reporting

- [x] 4. Implement analytics service
- [x] 4.1 Create analytics calculation functions
  - Write `get_open_rate(newsletter_id)` function
  - Write `get_click_rate(article_url)` function
  - Write `get_newsletter_metrics(newsletter_id)` function for overall stats
  - Write `get_article_performance(article_url)` function
  - _Requirements: 3.1, 3.2, 3.3, 3.4_

- [x] 4.2 Implement subscriber activity classification
  - Write `get_subscriber_activity_status(email)` function
  - Classify as 'active' (opened in last 30 days)
  - Classify as 'inactive' (no opens in 30-60 days)
  - Classify as 'dormant' (no opens in 60+ days)
  - Write `update_subscriber_activity_statuses()` batch function
  - _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5_

- [x] 4.3 Create analytics API endpoints
  - Create `GET /api/analytics/newsletter/<newsletter_id>` endpoint
  - Create `GET /api/analytics/article/<article_id>` endpoint
  - Create `GET /api/analytics/subscriber/<email>` endpoint
  - Create `POST /api/analytics/update-activity` endpoint
  - Return JSON with engagement metrics
  - _Requirements: 3.1, 3.2, 3.4, 5.4_

- [x] 4.4 Write unit tests for analytics
  - Test open rate calculation
  - Test click rate calculation
  - Test activity status classification
  - Test edge cases (no opens, no clicks)
  - _Requirements: 3.3, 3.4, 5.1, 5.2, 5.3_

## Phase 4: Privacy and Compliance

- [x] 5. Implement privacy features
- [x] 5.1 Create data anonymization function
  - Write function to anonymize tracking data older than 90 days
  - Remove email addresses from old records
  - Keep aggregated metrics
  - Create scheduled task to run daily
  - _Requirements: 4.2_

- [x] 5.2 Implement user data deletion
  - Create `DELETE /api/tracking/subscriber/<email>` endpoint
  - Delete all tracking records for subscriber
  - Delete from `newsletter_sends`, `link_clicks`, `subscriber_activity`
  - Return confirmation response
  - _Requirements: 4.3, 4.5_

- [x] 5.3 Add tracking opt-out support
  - Add `tracking_enabled` field to subscribers collection
  - Check opt-out status before creating tracking records
  - Skip tracking for opted-out subscribers
  - Update newsletter sending to respect opt-out
  - _Requirements: 4.4_

- [x] 5.4 Create anonymization endpoint
  - Create `POST /api/tracking/anonymize` endpoint
  - Trigger anonymization of old data
  - Return count of anonymized records
  - Add authentication/authorization
  - _Requirements: 4.2_

- [x] 5.5 Write privacy compliance tests
  - Test data anonymization works correctly
  - Test user data deletion removes all records
  - Test opt-out prevents tracking
  - Test anonymization preserves aggregated metrics
  - _Requirements: 4.2, 4.3, 4.4, 4.5_

## Phase 5: Configuration and Documentation

- [x] 6. Add configuration and environment setup
  - Add `TRACKING_ENABLED` to environment variables
  - Add `TRACKING_API_URL` configuration
  - Add `TRACKING_DATA_RETENTION_DAYS` configuration
  - Update `.env.template` with tracking variables
  - Update configuration documentation
  - _Requirements: All_

- [x] 7. Update database schema documentation
  - Document `newsletter_sends` collection schema
  - Document `link_clicks` collection schema
  - Document `subscriber_activity` collection schema
  - Add indexes documentation
  - Update DATABASE_SCHEMA.md
  - _Requirements: 1.3, 2.4, 5.4_

- [x] 8. Create tracking usage documentation
  - Document how to enable/disable tracking
  - Document analytics API endpoints
  - Document privacy features
  - Add examples of querying tracking data
  - Create README for tracking system
  - _Requirements: All_