dongho/Munich-news
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

slight update

Browse Source
This commit is contained in:
Dongho Kim
2025-11-11 14:34:43 +01:00
parent 760a458e66
commit f23f4b71d8
7 changed files with 748 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files

407
.kiro/specs/email-tracking/design.md Normal file
View File

@@ -0,0 +1,407 @@
# Email Tracking System Design
## Overview
The email tracking system enables Munich News Daily to measure subscriber engagement through email opens and link clicks. The system uses industry-standard techniques (tracking pixels and redirect URLs) while maintaining privacy compliance and performance.
## Architecture
### High-Level Components
```
┌─────────────────────────────────────────────────────────────┐
│ Newsletter System │
│ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Sender │─────▶│ Tracking │ │
│ │ Service │ │ Generator │ │
│ └──────────────┘ └──────────────┘ │
│ │ │ │
│ │ ▼ │
│ │ ┌──────────────┐ │
│ │ │ MongoDB │ │
│ │ │ (tracking) │ │
│ │ └──────────────┘ │
│ ▼ │
│ ┌──────────────┐ │
│ │ Email │ │
│ │ Client │ │
│ └──────────────┘ │
└─────────────────────────────────────────────────────────────┘
│ ▲
│ │
▼ │
┌─────────────────────────────────────────────────────────────┐
│ Backend API Server │
│ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Pixel │ │ Link │ │
│ │ Endpoint │ │ Redirect │ │
│ └──────────────┘ └──────────────┘ │
│ │ │ │
│ └──────────┬───────────┘ │
│ ▼ │
│ ┌──────────────┐ │
│ │ MongoDB │ │
│ │ (tracking) │ │
│ └──────────────┘ │
└─────────────────────────────────────────────────────────────┘
```
### Technology Stack
- **Backend**: Flask (Python) - existing backend server
- **Database**: MongoDB - existing database with new collections
- **Email**: SMTP (existing sender service)
- **Tracking**: UUID-based unique identifiers
- **Image**: 1x1 transparent PNG (base64 encoded)
## Components and Interfaces
### 1. Tracking ID Generator
**Purpose**: Generate unique tracking identifiers for emails and links
**Module**: `backend/services/tracking_service.py`
**Functions**:
```python
def generate_tracking_id() -> str:
"""Generate a unique tracking ID using UUID4"""
return str(uuid.uuid4())
def create_newsletter_tracking(newsletter_id: str, subscriber_email: str) -> dict:
"""Create tracking record for a newsletter send"""
# Returns tracking document with IDs for pixel and links
```
### 2. Tracking Pixel Endpoint
**Purpose**: Serve 1x1 transparent PNG and log email opens
**Endpoint**: `GET /api/track/pixel/<tracking_id>`
**Flow**:
1. Receive request with tracking_id
2. Look up tracking record in database
3. Log open event (email, timestamp, user-agent)
4. Return 1x1 transparent PNG image
5. Handle multiple opens (update last_opened_at)
**Response**:
- Status: 200 OK
- Content-Type: image/png
- Body: 1x1 transparent PNG (43 bytes)
### 3. Link Tracking Endpoint
**Purpose**: Track link clicks and redirect to original URL
**Endpoint**: `GET /api/track/click/<tracking_id>`
**Flow**:
1. Receive request with tracking_id
2. Look up tracking record and original URL
3. Log click event (email, article_url, timestamp, user-agent)
4. Redirect to original article URL (302 redirect)
5. Handle errors gracefully (redirect to homepage if invalid)
**Response**:
- Status: 302 Found
- Location: Original article URL
- Performance: < 200ms redirect time
### 4. Newsletter Template Modifier
**Purpose**: Inject tracking pixel and replace article links
**Module**: `news_sender/tracking_integration.py`
**Functions**:
```python
def inject_tracking_pixel(html: str, tracking_id: str, api_url: str) -> str:
"""Inject tracking pixel before closing </body> tag"""
pixel_url = f"{api_url}/api/track/pixel/{tracking_id}"
pixel_html = f'<img src="{pixel_url}" width="1" height="1" alt="" />'
return html.replace('</body>', f'{pixel_html}</body>')
def replace_article_links(html: str, articles: list, tracking_map: dict, api_url: str) -> str:
"""Replace article links with tracking URLs"""
# For each article link, replace with tracking URL
```
### 5. Analytics Service
**Purpose**: Calculate engagement metrics and identify active users
**Module**: `backend/services/analytics_service.py`
**Functions**:
```python
def get_open_rate(newsletter_id: str) -> float:
"""Calculate percentage of subscribers who opened"""
def get_click_rate(article_url: str) -> float:
"""Calculate percentage of subscribers who clicked"""
def get_subscriber_activity_status(email: str) -> str:
"""Return 'active', 'inactive', or 'dormant'"""
def update_subscriber_activity_statuses():
"""Batch update all subscriber activity statuses"""
```
## Data Models
### Newsletter Sends Collection (`newsletter_sends`)
Tracks each newsletter sent to each subscriber.
```javascript
{
_id: ObjectId,
newsletter_id: String, // Unique ID for this newsletter batch (date-based)
subscriber_email: String, // Recipient email
tracking_id: String, // Unique tracking ID for this send (UUID)
sent_at: DateTime, // When email was sent
opened: Boolean, // Whether email was opened
first_opened_at: DateTime, // First open timestamp (null if not opened)
last_opened_at: DateTime, // Most recent open timestamp
open_count: Number, // Number of times opened
created_at: DateTime // Record creation time
}
```
**Indexes**:
- `tracking_id` (unique) - Fast lookup for pixel requests
- `newsletter_id` - Analytics queries
- `subscriber_email` - User activity queries
- `sent_at` - Time-based queries
### Link Clicks Collection (`link_clicks`)
Tracks individual link clicks.
```javascript
{
_id: ObjectId,
tracking_id: String, // Unique tracking ID for this link (UUID)
newsletter_id: String, // Which newsletter this link was in
subscriber_email: String, // Who clicked
article_url: String, // Original article URL
article_title: String, // Article title for reporting
clicked_at: DateTime, // When link was clicked
user_agent: String, // Browser/client info
created_at: DateTime // Record creation time
}
```
**Indexes**:
- `tracking_id` (unique) - Fast lookup for redirect requests
- `newsletter_id` - Analytics queries
- `article_url` - Article performance queries
- `subscriber_email` - User activity queries
### Subscriber Activity Collection (`subscriber_activity`)
Aggregated activity status for each subscriber.
```javascript
{
_id: ObjectId,
email: String, // Subscriber email (unique)
status: String, // 'active', 'inactive', or 'dormant'
last_opened_at: DateTime, // Most recent email open
last_clicked_at: DateTime, // Most recent link click
total_opens: Number, // Lifetime open count
total_clicks: Number, // Lifetime click count
newsletters_received: Number, // Total newsletters sent
newsletters_opened: Number, // Total newsletters opened
updated_at: DateTime // Last status update
}
```
**Indexes**:
- `email` (unique) - Fast lookup
- `status` - Filter by activity level
- `last_opened_at` - Time-based queries
## Error Handling
### Tracking Pixel Failures
- **Invalid tracking_id**: Return 1x1 transparent PNG anyway (don't break email rendering)
- **Database error**: Log error, return pixel (fail silently)
- **Multiple opens**: Update existing record, don't create duplicate
### Link Redirect Failures
- **Invalid tracking_id**: Redirect to website homepage
- **Database error**: Log error, redirect to homepage
- **Missing original URL**: Redirect to homepage
### Privacy Compliance
- **Data retention**: Anonymize tracking data after 90 days
- Remove email addresses
- Keep aggregated metrics
- **Opt-out**: Check subscriber preferences before tracking
- **GDPR deletion**: Provide endpoint to delete all tracking data for a user
## Testing Strategy
### Unit Tests
1. **Tracking ID Generation**
- Test UUID format
- Test uniqueness
2. **Pixel Endpoint**
- Test valid tracking_id returns PNG
- Test invalid tracking_id returns PNG
- Test database logging
3. **Link Redirect**
- Test valid tracking_id redirects correctly
- Test invalid tracking_id redirects to homepage
- Test click logging
4. **Analytics Calculations**
- Test open rate calculation
- Test click rate calculation
- Test activity status classification
### Integration Tests
1. **End-to-End Newsletter Flow**
- Send newsletter with tracking
- Simulate email open (pixel request)
- Simulate link click
- Verify database records
2. **Privacy Compliance**
- Test data anonymization
- Test user data deletion
- Test opt-out handling
### Performance Tests
1. **Redirect Speed**
- Measure redirect time (target: < 200ms)
- Test under load (100 concurrent requests)
2. **Pixel Serving**
- Test pixel response time
- Test caching headers
## API Endpoints
### Tracking Endpoints
```
GET /api/track/pixel/<tracking_id>
- Returns: 1x1 transparent PNG
- Logs: Email open event
GET /api/track/click/<tracking_id>
- Returns: 302 redirect to article URL
- Logs: Link click event
```
### Analytics Endpoints
```
GET /api/analytics/newsletter/<newsletter_id>
- Returns: Open rate, click rate, engagement metrics
GET /api/analytics/article/<article_id>
- Returns: Click count, click rate for specific article
GET /api/analytics/subscriber/<email>
- Returns: Activity status, engagement history
POST /api/analytics/update-activity
- Triggers: Batch update of subscriber activity statuses
- Returns: Update count
```
### Privacy Endpoints
```
DELETE /api/tracking/subscriber/<email>
- Deletes: All tracking data for subscriber
- Returns: Deletion confirmation
POST /api/tracking/anonymize
- Triggers: Anonymize tracking data older than 90 days
- Returns: Anonymization count
```
## Implementation Phases
### Phase 1: Core Tracking (MVP)
- Tracking ID generation
- Pixel endpoint
- Link redirect endpoint
- Database collections
- Newsletter template integration
### Phase 2: Analytics
- Open rate calculation
- Click rate calculation
- Activity status classification
- Analytics API endpoints
### Phase 3: Privacy & Compliance
- Data anonymization
- User data deletion
- Opt-out handling
- Privacy notices
### Phase 4: Optimization
- Caching for pixel endpoint
- Performance monitoring
- Batch processing for activity updates
## Security Considerations
1. **Rate Limiting**: Prevent abuse of tracking endpoints
2. **Input Validation**: Validate all tracking_ids (UUID format)
3. **SQL Injection**: Use parameterized queries (MongoDB safe by default)
4. **Privacy**: Don't expose subscriber emails in URLs
5. **HTTPS**: Ensure all tracking URLs use HTTPS in production
## Configuration
Add to `backend/.env`:
```env
# Tracking Configuration
TRACKING_ENABLED=true
TRACKING_API_URL=http://localhost:5000
TRACKING_DATA_RETENTION_DAYS=90
```
## Monitoring and Metrics
### Key Metrics to Track
1. **Email Opens**
- Overall open rate
- Open rate by newsletter
- Time to first open
2. **Link Clicks**
- Overall click rate
- Click rate by article
- Click-through rate (CTR)
3. **Subscriber Engagement**
- Active subscriber count
- Inactive subscriber count
- Dormant subscriber count
4. **System Performance**
- Pixel response time
- Redirect response time
- Database query performance

77
.kiro/specs/email-tracking/requirements.md Normal file
View File

@@ -0,0 +1,77 @@
# Requirements Document
## Introduction
This document outlines the requirements for implementing email tracking functionality in the Munich News Daily newsletter system. The system will track email opens and link clicks to measure subscriber engagement and identify active users.
## Glossary
- **Newsletter System**: The Munich News Daily email sending service
- **Tracking Pixel**: A 1x1 transparent image embedded in emails to detect opens
- **Tracking Link**: A redirecting URL that logs clicks before sending users to the actual destination
- **Subscriber**: A user who receives the newsletter
- **Email Open**: When a subscriber's email client loads the tracking pixel
- **Link Click**: When a subscriber clicks a tracked link in the newsletter
- **Engagement Metrics**: Data about subscriber interactions with newsletters
## Requirements
### Requirement 1: Track Email Opens
**User Story:** As a newsletter administrator, I want to track when subscribers open emails, so that I can measure engagement and identify active users.
#### Acceptance Criteria
1. WHEN a newsletter is sent, THE Newsletter System SHALL embed a unique tracking pixel in each email
2. WHEN a subscriber opens the email, THE Newsletter System SHALL record the open event with timestamp
3. THE Newsletter System SHALL store the subscriber email, newsletter ID, and open timestamp in the database
4. THE Newsletter System SHALL serve the tracking pixel as a 1x1 transparent PNG image
5. THE Newsletter System SHALL handle multiple opens from the same subscriber for the same newsletter
### Requirement 2: Track Link Clicks
**User Story:** As a newsletter administrator, I want to track when subscribers click on article links, so that I can understand which content is most engaging.
#### Acceptance Criteria
1. WHEN a newsletter is generated, THE Newsletter System SHALL replace all article links with unique tracking URLs
2. WHEN a subscriber clicks a tracking URL, THE Newsletter System SHALL record the click event with timestamp
3. WHEN a click is recorded, THE Newsletter System SHALL redirect the subscriber to the original article URL
4. THE Newsletter System SHALL store the subscriber email, article link, and click timestamp in the database
5. THE Newsletter System SHALL complete the redirect within 200 milliseconds
### Requirement 3: Generate Engagement Reports
**User Story:** As a newsletter administrator, I want to view engagement metrics, so that I can understand subscriber behavior and content performance.
#### Acceptance Criteria
1. THE Newsletter System SHALL provide an API endpoint to retrieve open rates by newsletter
2. THE Newsletter System SHALL provide an API endpoint to retrieve click rates by article
3. THE Newsletter System SHALL calculate the percentage of subscribers who opened each newsletter
4. THE Newsletter System SHALL calculate the percentage of subscribers who clicked each article link
5. THE Newsletter System SHALL identify subscribers who have not opened emails in the last 30 days
### Requirement 4: Privacy and Compliance
**User Story:** As a newsletter administrator, I want to respect subscriber privacy, so that the system complies with privacy regulations.
#### Acceptance Criteria
1. THE Newsletter System SHALL include a privacy notice in the newsletter footer explaining tracking
2. THE Newsletter System SHALL anonymize tracking data after 90 days by removing email addresses
3. THE Newsletter System SHALL provide an API endpoint to delete all tracking data for a specific subscriber
4. THE Newsletter System SHALL not track subscribers who have opted out of tracking
5. WHERE a subscriber unsubscribes, THE Newsletter System SHALL delete all their tracking data
### Requirement 5: Identify Active Users
**User Story:** As a newsletter administrator, I want to identify active subscribers, so that I can segment my audience and improve targeting.
#### Acceptance Criteria
1. THE Newsletter System SHALL classify a subscriber as "active" if they opened an email in the last 30 days
2. THE Newsletter System SHALL classify a subscriber as "inactive" if they have not opened an email in 30-60 days
3. THE Newsletter System SHALL classify a subscriber as "dormant" if they have not opened an email in over 60 days
4. THE Newsletter System SHALL provide an API endpoint to retrieve subscriber activity status
5. THE Newsletter System SHALL update subscriber activity status daily

170
.kiro/specs/email-tracking/tasks.md Normal file
View File

@@ -0,0 +1,170 @@
# Implementation Plan
## Phase 1: Core Tracking Infrastructure
- [x] 1. Set up database collections and indexes
- Create MongoDB collections: `newsletter_sends`, `link_clicks`, `subscriber_activity`
- Add indexes for performance: `tracking_id` (unique), `newsletter_id`, `subscriber_email`, `sent_at`
- Write database initialization script
- _Requirements: 1.3, 2.4_
- [x] 2. Implement tracking service
- [x] 2.1 Create tracking ID generator
- Write `generate_tracking_id()` function using UUID4
- Write `create_newsletter_tracking()` function to create tracking records
- Add configuration for tracking enable/disable
- _Requirements: 1.1, 2.1_
- [x] 2.2 Implement tracking pixel endpoint
- Create Flask route `GET /api/track/pixel/<tracking_id>`
- Generate 1x1 transparent PNG (base64 encoded)
- Log email open event to `newsletter_sends` collection
- Handle multiple opens (update `last_opened_at` and `open_count`)
- Return PNG with proper headers (Content-Type: image/png)
- _Requirements: 1.2, 1.3, 1.4, 1.5_
- [x] 2.3 Implement link redirect endpoint
- Create Flask route `GET /api/track/click/<tracking_id>`
- Look up original article URL from tracking record
- Log click event to `link_clicks` collection
- Redirect to original URL with 302 status
- Handle invalid tracking_id (redirect to homepage)
- Ensure redirect completes within 200ms
- _Requirements: 2.2, 2.3, 2.4, 2.5_
- [x] 2.4 Write unit tests for tracking endpoints
- Test pixel endpoint returns PNG for valid tracking_id
- Test pixel endpoint returns PNG for invalid tracking_id (fail silently)
- Test link redirect works correctly
- Test link redirect handles invalid tracking_id
- Test database logging for opens and clicks
- _Requirements: 1.2, 1.4, 2.2, 2.3_
## Phase 2: Newsletter Integration
- [x] 3. Integrate tracking into sender service
- [x] 3.1 Create tracking integration module
- Write `inject_tracking_pixel()` function to add pixel to HTML
- Write `replace_article_links()` function to replace links with tracking URLs
- Write `generate_tracking_urls()` function to create tracking records for all links
- Add tracking configuration to sender service
- _Requirements: 1.1, 2.1_
- [x] 3.2 Modify newsletter sending flow
- Update `send_newsletter()` to generate tracking IDs for each subscriber
- Create tracking records in database before sending
- Inject tracking pixel into newsletter HTML
- Replace article links with tracking URLs
- Store newsletter_id and tracking metadata
- _Requirements: 1.1, 1.3, 2.1, 2.4_
- [x] 3.3 Update newsletter template
- Ensure template supports dynamic tracking pixel injection
- Ensure article links are properly structured for replacement
- Add privacy notice to footer about tracking
- _Requirements: 4.1_
- [x] 3.4 Test newsletter with tracking
- Send test newsletter with tracking enabled
- Verify tracking pixel is embedded correctly
- Verify article links are replaced with tracking URLs
- Test email open tracking works
- Test link click tracking works
- _Requirements: 1.1, 1.2, 2.1, 2.2_
## Phase 3: Analytics and Reporting
- [x] 4. Implement analytics service
- [x] 4.1 Create analytics calculation functions
- Write `get_open_rate(newsletter_id)` function
- Write `get_click_rate(article_url)` function
- Write `get_newsletter_metrics(newsletter_id)` function for overall stats
- Write `get_article_performance(article_url)` function
- _Requirements: 3.1, 3.2, 3.3, 3.4_
- [x] 4.2 Implement subscriber activity classification
- Write `get_subscriber_activity_status(email)` function
- Classify as 'active' (opened in last 30 days)
- Classify as 'inactive' (no opens in 30-60 days)
- Classify as 'dormant' (no opens in 60+ days)
- Write `update_subscriber_activity_statuses()` batch function
- _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5_
- [x] 4.3 Create analytics API endpoints
- Create `GET /api/analytics/newsletter/<newsletter_id>` endpoint
- Create `GET /api/analytics/article/<article_id>` endpoint
- Create `GET /api/analytics/subscriber/<email>` endpoint
- Create `POST /api/analytics/update-activity` endpoint
- Return JSON with engagement metrics
- _Requirements: 3.1, 3.2, 3.4, 5.4_
- [x] 4.4 Write unit tests for analytics
- Test open rate calculation
- Test click rate calculation
- Test activity status classification
- Test edge cases (no opens, no clicks)
- _Requirements: 3.3, 3.4, 5.1, 5.2, 5.3_
## Phase 4: Privacy and Compliance
- [x] 5. Implement privacy features
- [x] 5.1 Create data anonymization function
- Write function to anonymize tracking data older than 90 days
- Remove email addresses from old records
- Keep aggregated metrics
- Create scheduled task to run daily
- _Requirements: 4.2_
- [x] 5.2 Implement user data deletion
- Create `DELETE /api/tracking/subscriber/<email>` endpoint
- Delete all tracking records for subscriber
- Delete from `newsletter_sends`, `link_clicks`, `subscriber_activity`
- Return confirmation response
- _Requirements: 4.3, 4.5_
- [x] 5.3 Add tracking opt-out support
- Add `tracking_enabled` field to subscribers collection
- Check opt-out status before creating tracking records
- Skip tracking for opted-out subscribers
- Update newsletter sending to respect opt-out
- _Requirements: 4.4_
- [x] 5.4 Create anonymization endpoint
- Create `POST /api/tracking/anonymize` endpoint
- Trigger anonymization of old data
- Return count of anonymized records
- Add authentication/authorization
- _Requirements: 4.2_
- [x] 5.5 Write privacy compliance tests
- Test data anonymization works correctly
- Test user data deletion removes all records
- Test opt-out prevents tracking
- Test anonymization preserves aggregated metrics
- _Requirements: 4.2, 4.3, 4.4, 4.5_
## Phase 5: Configuration and Documentation
- [x] 6. Add configuration and environment setup
- Add `TRACKING_ENABLED` to environment variables
- Add `TRACKING_API_URL` configuration
- Add `TRACKING_DATA_RETENTION_DAYS` configuration
- Update `.env.template` with tracking variables
- Update configuration documentation
- _Requirements: All_
- [x] 7. Update database schema documentation
- Document `newsletter_sends` collection schema
- Document `link_clicks` collection schema
- Document `subscriber_activity` collection schema
- Add indexes documentation
- Update DATABASE_SCHEMA.md
- _Requirements: 1.3, 2.4, 5.4_
- [x] 8. Create tracking usage documentation
- Document how to enable/disable tracking
- Document analytics API endpoints
- Document privacy features
- Add examples of querying tracking data
- Create README for tracking system
- _Requirements: All_