Munich-news/.kiro/specs/email-tracking/tasks.md

Implementation Plan

Phase 1: Core Tracking Infrastructure

• 1. Set up database collections and indexes

  • Create MongoDB collections: newsletter_sends, link_clicks, subscriber_activity
  • Add indexes for performance: tracking_id (unique), newsletter_id, subscriber_email, sent_at
  • Write database initialization script
  • Requirements: 1.3, 2.4

• 2. Implement tracking service

• 2.1 Create tracking ID generator

  • Write generate_tracking_id() function using UUID4
  • Write create_newsletter_tracking() function to create tracking records
  • Add configuration for tracking enable/disable
  • Requirements: 1.1, 2.1

• 2.2 Implement tracking pixel endpoint

  • Create Flask route GET /api/track/pixel/<tracking_id>
  • Generate 1x1 transparent PNG (base64 encoded)
  • Log email open event to newsletter_sends collection
  • Handle multiple opens (update last_opened_at and open_count)
  • Return PNG with proper headers (Content-Type: image/png)
  • Requirements: 1.2, 1.3, 1.4, 1.5

• 2.3 Implement link redirect endpoint

  • Create Flask route GET /api/track/click/<tracking_id>
  • Look up original article URL from tracking record
  • Log click event to link_clicks collection
  • Redirect to original URL with 302 status
  • Handle invalid tracking_id (redirect to homepage)
  • Ensure redirect completes within 200ms
  • Requirements: 2.2, 2.3, 2.4, 2.5

• 2.4 Write unit tests for tracking endpoints

  • Test pixel endpoint returns PNG for valid tracking_id
  • Test pixel endpoint returns PNG for invalid tracking_id (fail silently)
  • Test link redirect works correctly
  • Test link redirect handles invalid tracking_id
  • Test database logging for opens and clicks
  • Requirements: 1.2, 1.4, 2.2, 2.3

Phase 2: Newsletter Integration

• 3. Integrate tracking into sender service

• 3.1 Create tracking integration module

  • Write inject_tracking_pixel() function to add pixel to HTML
  • Write replace_article_links() function to replace links with tracking URLs
  • Write generate_tracking_urls() function to create tracking records for all links
  • Add tracking configuration to sender service
  • Requirements: 1.1, 2.1

• 3.2 Modify newsletter sending flow

  • Update send_newsletter() to generate tracking IDs for each subscriber
  • Create tracking records in database before sending
  • Inject tracking pixel into newsletter HTML
  • Replace article links with tracking URLs
  • Store newsletter_id and tracking metadata
  • Requirements: 1.1, 1.3, 2.1, 2.4

• 3.3 Update newsletter template

  • Ensure template supports dynamic tracking pixel injection
  • Ensure article links are properly structured for replacement
  • Add privacy notice to footer about tracking
  • Requirements: 4.1

• 3.4 Test newsletter with tracking

  • Send test newsletter with tracking enabled
  • Verify tracking pixel is embedded correctly
  • Verify article links are replaced with tracking URLs
  • Test email open tracking works
  • Test link click tracking works
  • Requirements: 1.1, 1.2, 2.1, 2.2

Phase 3: Analytics and Reporting

• 4. Implement analytics service

• 4.1 Create analytics calculation functions

  • Write get_open_rate(newsletter_id) function
  • Write get_click_rate(article_url) function
  • Write get_newsletter_metrics(newsletter_id) function for overall stats
  • Write get_article_performance(article_url) function
  • Requirements: 3.1, 3.2, 3.3, 3.4

• 4.2 Implement subscriber activity classification

  • Write get_subscriber_activity_status(email) function
  • Classify as 'active' (opened in last 30 days)
  • Classify as 'inactive' (no opens in 30-60 days)
  • Classify as 'dormant' (no opens in 60+ days)
  • Write update_subscriber_activity_statuses() batch function
  • Requirements: 5.1, 5.2, 5.3, 5.4, 5.5

• 4.3 Create analytics API endpoints

  • Create GET /api/analytics/newsletter/<newsletter_id> endpoint
  • Create GET /api/analytics/article/<article_id> endpoint
  • Create GET /api/analytics/subscriber/<email> endpoint
  • Create POST /api/analytics/update-activity endpoint
  • Return JSON with engagement metrics
  • Requirements: 3.1, 3.2, 3.4, 5.4

• 4.4 Write unit tests for analytics

  • Test open rate calculation
  • Test click rate calculation
  • Test activity status classification
  • Test edge cases (no opens, no clicks)
  • Requirements: 3.3, 3.4, 5.1, 5.2, 5.3

Phase 4: Privacy and Compliance

• 5. Implement privacy features

• 5.1 Create data anonymization function

  • Write function to anonymize tracking data older than 90 days
  • Remove email addresses from old records
  • Keep aggregated metrics
  • Create scheduled task to run daily
  • Requirements: 4.2

• 5.2 Implement user data deletion

  • Create DELETE /api/tracking/subscriber/<email> endpoint
  • Delete all tracking records for subscriber
  • Delete from newsletter_sends, link_clicks, subscriber_activity
  • Return confirmation response
  • Requirements: 4.3, 4.5

• 5.3 Add tracking opt-out support

  • Add tracking_enabled field to subscribers collection
  • Check opt-out status before creating tracking records
  • Skip tracking for opted-out subscribers
  • Update newsletter sending to respect opt-out
  • Requirements: 4.4

• 5.4 Create anonymization endpoint

  • Create POST /api/tracking/anonymize endpoint
  • Trigger anonymization of old data
  • Return count of anonymized records
  • Add authentication/authorization
  • Requirements: 4.2

• 5.5 Write privacy compliance tests

  • Test data anonymization works correctly
  • Test user data deletion removes all records
  • Test opt-out prevents tracking
  • Test anonymization preserves aggregated metrics
  • Requirements: 4.2, 4.3, 4.4, 4.5

Phase 5: Configuration and Documentation

• 6. Add configuration and environment setup

  • Add TRACKING_ENABLED to environment variables
  • Add TRACKING_API_URL configuration
  • Add TRACKING_DATA_RETENTION_DAYS configuration
  • Update .env.template with tracking variables
  • Update configuration documentation
  • Requirements: All

• 7. Update database schema documentation

  • Document newsletter_sends collection schema
  • Document link_clicks collection schema
  • Document subscriber_activity collection schema
  • Add indexes documentation
  • Update DATABASE_SCHEMA.md
  • Requirements: 1.3, 2.4, 5.4

• 8. Create tracking usage documentation

  • Document how to enable/disable tracking
  • Document analytics API endpoints
  • Document privacy features
  • Add examples of querying tracking data
  • Create README for tracking system
  • Requirements: All