dongho/Munich-news
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Dongho Kim
2025-11-10 19:13:33 +01:00
commit ac5738c29d
64 changed files with 9445 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
news_sender/.gitignore vendored Normal file
View File

@@ -0,0 +1,28 @@
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
env/
venv/
ENV/
.venv
# Environment variables
.env
.env.local
# Generated files
newsletter_preview.html
# IDE
.vscode/
.idea/
*.swp
*.swo
*~
# OS
.DS_Store
Thumbs.db

303
news_sender/README.md Normal file
View File

@@ -0,0 +1,303 @@
# News Sender Microservice
Standalone service for sending Munich News Daily newsletters to subscribers.
## Features
- 📧 Sends beautiful HTML newsletters
- 🤖 Uses AI-generated article summaries
- 📊 Tracks sending statistics
- 🧪 Test mode for development
- 📝 Preview generation
- 🔄 Fetches data from shared MongoDB
## Installation
```bash
cd news_sender
pip install -r requirements.txt
```
## Configuration
The service uses the same `.env` file as the backend (`../backend/.env`):
```env
# MongoDB
MONGODB_URI=mongodb://localhost:27017/
# Email (Gmail example)
SMTP_SERVER=smtp.gmail.com
SMTP_PORT=587
EMAIL_USER=your-email@gmail.com
EMAIL_PASSWORD=your-app-password
# Newsletter Settings (optional)
NEWSLETTER_MAX_ARTICLES=10
WEBSITE_URL=http://localhost:3000
```
**Gmail Setup:**
1. Enable 2-factor authentication
2. Generate an App Password: https://support.google.com/accounts/answer/185833
3. Use the App Password (not your regular password)
## Usage
### 1. Preview Newsletter
Generate HTML preview without sending:
```bash
python sender_service.py preview
```
This creates `newsletter_preview.html` - open it in your browser to see how the newsletter looks.
### 2. Send Test Email
Send to a single email address for testing:
```bash
python sender_service.py test your-email@example.com
```
### 3. Send to All Subscribers
Send newsletter to all active subscribers:
```bash
# Send with default article count (10)
python sender_service.py send
# Send with custom article count
python sender_service.py send 15
```
### 4. Use as Python Module
```python
from sender_service import send_newsletter, preview_newsletter
# Send newsletter
result = send_newsletter(max_articles=10)
print(f"Sent to {result['sent_count']} subscribers")
# Generate preview
html = preview_newsletter(max_articles=5)
```
## How It Works
```
┌─────────────────────────────────────────────────────────┐
│ 1. Fetch Articles from MongoDB │
│ - Get latest articles with AI summaries │
│ - Sort by creation date (newest first) │
└─────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ 2. Fetch Active Subscribers │
│ - Get all subscribers with status='active' │
└─────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ 3. Render Newsletter HTML │
│ - Load newsletter_template.html │
│ - Populate with articles and metadata │
│ - Generate beautiful HTML email │
└─────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ 4. Send Emails │
│ - Connect to SMTP server │
│ - Send to each subscriber │
│ - Track success/failure │
└─────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ 5. Report Statistics │
│ - Total sent │
│ - Failed sends │
│ - Error details │
└─────────────────────────────────────────────────────────┘
```
## Output Example
```
======================================================================
📧 Munich News Daily - Newsletter Sender
======================================================================
Fetching latest 10 articles with AI summaries...
✓ Found 10 articles
Fetching active subscribers...
✓ Found 150 active subscriber(s)
Rendering newsletter HTML...
✓ Newsletter rendered
Sending newsletter: 'Munich News Daily - November 10, 2024'
----------------------------------------------------------------------
[1/150] Sending to user1@example.com... ✓
[2/150] Sending to user2@example.com... ✓
[3/150] Sending to user3@example.com... ✓
...
======================================================================
📊 Sending Complete
======================================================================
✓ Successfully sent: 148
✗ Failed: 2
📰 Articles included: 10
======================================================================
```
## Scheduling
### Using Cron (Linux/Mac)
Send newsletter daily at 8 AM:
```bash
# Edit crontab
crontab -e
# Add this line
0 8 * * * cd /path/to/news_sender && /path/to/venv/bin/python sender_service.py send
```
### Using systemd Timer (Linux)
Create `/etc/systemd/system/news-sender.service`:
```ini
[Unit]
Description=Munich News Sender
[Service]
Type=oneshot
WorkingDirectory=/path/to/news_sender
ExecStart=/path/to/venv/bin/python sender_service.py send
User=your-user
```
Create `/etc/systemd/system/news-sender.timer`:
```ini
[Unit]
Description=Send Munich News Daily at 8 AM
[Timer]
OnCalendar=daily
OnCalendar=*-*-* 08:00:00
[Install]
WantedBy=timers.target
```
Enable and start:
```bash
sudo systemctl enable news-sender.timer
sudo systemctl start news-sender.timer
```
### Using Docker
Create `Dockerfile`:
```dockerfile
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY sender_service.py newsletter_template.html ./
CMD ["python", "sender_service.py", "send"]
```
Build and run:
```bash
docker build -t news-sender .
docker run --env-file ../backend/.env news-sender
```
## Troubleshooting
### "Email credentials not configured"
- Check that `EMAIL_USER` and `EMAIL_PASSWORD` are set in `.env`
- For Gmail, use an App Password, not your regular password
### "No articles with summaries found"
- Run the crawler first: `cd ../news_crawler && python crawler_service.py 10`
- Make sure Ollama is enabled and working
- Check MongoDB has articles with `summary` field
### "No active subscribers found"
- Add subscribers via the backend API
- Check subscriber status is 'active' in MongoDB
### SMTP Connection Errors
- Verify SMTP server and port are correct
- Check firewall isn't blocking SMTP port
- For Gmail, ensure "Less secure app access" is enabled or use App Password
### Emails Going to Spam
- Set up SPF, DKIM, and DMARC records for your domain
- Use a verified email address
- Avoid spam trigger words in subject/content
- Include unsubscribe link (already included in template)
## Architecture
This is a standalone microservice that:
- Runs independently of the backend
- Shares the same MongoDB database
- Can be deployed separately
- Can be scheduled independently
- Has no dependencies on backend code
## Integration with Other Services
```
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Backend │ │ Crawler │ │ Sender │
│ (Flask) │ │ (Scraper) │ │ (Email) │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
│ │ │
│ │ │
└────────────────────┴─────────────────────┘
│
┌───────▼────────┐
│ MongoDB │
│ (Shared DB) │
└────────────────┘
```
## Next Steps
1. **Test the newsletter:**
```bash
python sender_service.py test your-email@example.com
```
2. **Schedule daily sending:**
- Set up cron job or systemd timer
- Choose appropriate time (e.g., 8 AM)
3. **Monitor sending:**
- Check logs for errors
- Track open rates (requires email tracking service)
- Monitor spam complaints
4. **Optimize:**
- Add email tracking pixels
- A/B test subject lines
- Personalize content per subscriber

162
news_sender/newsletter_template.html Normal file
View File

@@ -0,0 +1,162 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Munich News Daily</title>
<!--[if mso]>
<style type="text/css">
body, table, td {font-family: Arial, Helvetica, sans-serif !important;}
</style>
<![endif]-->
</head>
<body style="margin: 0; padding: 0; background-color: #f4f4f4; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, sans-serif;">
<!-- Wrapper Table -->
<table role="presentation" width="100%" cellpadding="0" cellspacing="0" border="0" style="background-color: #f4f4f4;">
<tr>
<td align="center" style="padding: 20px 0;">
<!-- Main Container -->
<table role="presentation" width="600" cellpadding="0" cellspacing="0" border="0" style="background-color: #ffffff; max-width: 600px;">
<!-- Header -->
<tr>
<td style="background-color: #1a1a1a; padding: 30px 40px; text-align: center;">
<h1 style="margin: 0 0 8px 0; font-size: 28px; font-weight: 700; color: #ffffff; letter-spacing: -0.5px;">
Munich News Daily
</h1>
<p style="margin: 0; font-size: 14px; color: #999999; letter-spacing: 0.5px;">
{{ date }}
</p>
</td>
</tr>
<!-- Greeting -->
<tr>
<td style="padding: 30px 40px 20px 40px;">
<p style="margin: 0; font-size: 16px; line-height: 1.5; color: #333333;">
Good morning ☀️
</p>
<p style="margin: 15px 0 0 0; font-size: 15px; line-height: 1.6; color: #666666;">
Here's what's happening in Munich today. We've summarized {{ article_count }} stories using AI so you can stay informed in under 5 minutes.
</p>
</td>
</tr>
<!-- Divider -->
<tr>
<td style="padding: 0 40px;">
<div style="height: 1px; background-color: #e0e0e0;"></div>
</td>
</tr>
<!-- Articles -->
{% for article in articles %}
<tr>
<td style="padding: 25px 40px;">
<!-- Article Number Badge -->
<table role="presentation" width="100%" cellpadding="0" cellspacing="0" border="0">
<tr>
<td>
<span style="display: inline-block; background-color: #000000; color: #ffffff; width: 24px; height: 24px; line-height: 24px; text-align: center; border-radius: 50%; font-size: 12px; font-weight: 600;">
{{ loop.index }}
</span>
</td>
</tr>
</table>
<!-- Article Title -->
<h2 style="margin: 12px 0 8px 0; font-size: 19px; font-weight: 700; line-height: 1.3; color: #1a1a1a;">
{{ article.title }}
</h2>
<!-- Article Meta -->
<p style="margin: 0 0 12px 0; font-size: 13px; color: #999999;">
<span style="color: #000000; font-weight: 600;">{{ article.source }}</span>
{% if article.author %}
<span> • {{ article.author }}</span>
{% endif %}
</p>
<!-- Article Summary -->
<p style="margin: 0 0 15px 0; font-size: 15px; line-height: 1.6; color: #333333;">
{{ article.summary }}
</p>
<!-- Read More Link -->
<a href="{{ article.link }}" style="display: inline-block; color: #000000; text-decoration: none; font-size: 14px; font-weight: 600; border-bottom: 2px solid #000000; padding-bottom: 2px;">
Read more →
</a>
</td>
</tr>
<!-- Article Divider -->
{% if not loop.last %}
<tr>
<td style="padding: 0 40px;">
<div style="height: 1px; background-color: #f0f0f0;"></div>
</td>
</tr>
{% endif %}
{% endfor %}
<!-- Bottom Divider -->
<tr>
<td style="padding: 25px 40px 0 40px;">
<div style="height: 1px; background-color: #e0e0e0;"></div>
</td>
</tr>
<!-- Summary Box -->
<tr>
<td style="padding: 30px 40px;">
<table role="presentation" width="100%" cellpadding="0" cellspacing="0" border="0" style="background-color: #f8f8f8; border-radius: 8px;">
<tr>
<td style="padding: 25px; text-align: center;">
<p style="margin: 0 0 8px 0; font-size: 13px; color: #666666; text-transform: uppercase; letter-spacing: 1px; font-weight: 600;">
Today's Digest
</p>
<p style="margin: 0; font-size: 36px; font-weight: 700; color: #000000;">
{{ article_count }}
</p>
<p style="margin: 8px 0 0 0; font-size: 14px; color: #666666;">
stories • AI-summarized • 5 min read
</p>
</td>
</tr>
</table>
</td>
</tr>
<!-- Footer -->
<tr>
<td style="background-color: #1a1a1a; padding: 30px 40px; text-align: center;">
<p style="margin: 0 0 15px 0; font-size: 14px; color: #ffffff; font-weight: 600;">
Munich News Daily
</p>
<p style="margin: 0 0 20px 0; font-size: 13px; color: #999999; line-height: 1.5;">
AI-powered news summaries for busy people.<br>
Delivered daily to your inbox.
</p>
<!-- Footer Links -->
<p style="margin: 0; font-size: 12px; color: #666666;">
<a href="{{ website_link }}" style="color: #999999; text-decoration: none;">Visit Website</a>
<span style="color: #444444;"> • </span>
<a href="{{ unsubscribe_link }}" style="color: #999999; text-decoration: none;">Unsubscribe</a>
</p>
<p style="margin: 20px 0 0 0; font-size: 11px; color: #666666;">
© {{ year }} Munich News Daily. All rights reserved.
</p>
</td>
</tr>
</table>
<!-- End Main Container -->
</td>
</tr>
</table>
<!-- End Wrapper Table -->
</body>
</html>

3
news_sender/requirements.txt Normal file
View File

@@ -0,0 +1,3 @@
pymongo==4.6.1
python-dotenv==1.0.0
Jinja2==3.1.2

313
news_sender/sender_service.py Normal file
View File

@@ -0,0 +1,313 @@
#!/usr/bin/env python
"""
News Sender Service - Standalone microservice for sending newsletters
Fetches articles from MongoDB and sends to subscribers via email
"""
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from datetime import datetime
from pathlib import Path
from jinja2 import Template
from pymongo import MongoClient
import os
from dotenv import load_dotenv
# Load environment variables from backend/.env
backend_dir = Path(__file__).parent.parent / 'backend'
env_path = backend_dir / '.env'
if env_path.exists():
load_dotenv(dotenv_path=env_path)
print(f"✓ Loaded configuration from: {env_path}")
else:
print(f"⚠ Warning: .env file not found at {env_path}")
class Config:
"""Configuration for news sender"""
# MongoDB
MONGODB_URI = os.getenv('MONGODB_URI', 'mongodb://localhost:27017/')
DB_NAME = 'munich_news'
# Email
SMTP_SERVER = os.getenv('SMTP_SERVER', 'smtp.gmail.com')
SMTP_PORT = int(os.getenv('SMTP_PORT', '587'))
EMAIL_USER = os.getenv('EMAIL_USER', '')
EMAIL_PASSWORD = os.getenv('EMAIL_PASSWORD', '')
# Newsletter
MAX_ARTICLES = int(os.getenv('NEWSLETTER_MAX_ARTICLES', '10'))
WEBSITE_URL = os.getenv('WEBSITE_URL', 'http://localhost:3000')
# MongoDB connection
client = MongoClient(Config.MONGODB_URI)
db = client[Config.DB_NAME]
articles_collection = db['articles']
subscribers_collection = db['subscribers']
def get_latest_articles(max_articles=10):
"""
Get latest articles with AI summaries from database
Returns:
list: Articles with summaries
"""
cursor = articles_collection.find(
{'summary': {'$exists': True, '$ne': None}}
).sort('created_at', -1).limit(max_articles)
articles = []
for doc in cursor:
articles.append({
'title': doc.get('title', ''),
'author': doc.get('author'),
'link': doc.get('link', ''),
'summary': doc.get('summary', ''),
'source': doc.get('source', ''),
'published_at': doc.get('published_at', '')
})
return articles
def get_active_subscribers():
"""
Get all active subscribers from database
Returns:
list: Email addresses of active subscribers
"""
cursor = subscribers_collection.find({'status': 'active'})
return [doc['email'] for doc in cursor]
def render_newsletter_html(articles):
"""
Render newsletter HTML from template
Args:
articles: List of article dictionaries
Returns:
str: Rendered HTML content
"""
# Load template
template_path = Path(__file__).parent / 'newsletter_template.html'
with open(template_path, 'r', encoding='utf-8') as f:
template_content = f.read()
template = Template(template_content)
# Prepare template data
now = datetime.now()
template_data = {
'date': now.strftime('%A, %B %d, %Y'),
'year': now.year,
'article_count': len(articles),
'articles': articles,
'unsubscribe_link': f'{Config.WEBSITE_URL}/unsubscribe',
'website_link': Config.WEBSITE_URL
}
# Render HTML
return template.render(**template_data)
def send_email(to_email, subject, html_content):
"""
Send email to a single recipient
Args:
to_email: Recipient email address
subject: Email subject
html_content: HTML content of email
Returns:
tuple: (success: bool, error: str or None)
"""
try:
msg = MIMEMultipart('alternative')
msg['Subject'] = subject
msg['From'] = f'Munich News Daily <{Config.EMAIL_USER}>'
msg['To'] = to_email
msg['Date'] = datetime.now().strftime('%a, %d %b %Y %H:%M:%S %z')
msg['Message-ID'] = f'<{datetime.now().timestamp()}.{to_email}@dongho.kim>'
msg['X-Mailer'] = 'Munich News Daily Sender'
# Add plain text version as fallback
plain_text = "This email requires HTML support. Please view it in an HTML-capable email client."
msg.attach(MIMEText(plain_text, 'plain', 'utf-8'))
# Add HTML version
msg.attach(MIMEText(html_content, 'html', 'utf-8'))
server = smtplib.SMTP(Config.SMTP_SERVER, Config.SMTP_PORT)
server.starttls()
server.login(Config.EMAIL_USER, Config.EMAIL_PASSWORD)
server.send_message(msg)
server.quit()
return True, None
except Exception as e:
return False, str(e)
def send_newsletter(max_articles=None, test_email=None):
"""
Send newsletter to all active subscribers
Args:
max_articles: Maximum number of articles to include (default from config)
test_email: If provided, send only to this email (for testing)
Returns:
dict: Statistics about sending
"""
print("\n" + "="*70)
print("📧 Munich News Daily - Newsletter Sender")
print("="*70)
# Validate email configuration
if not Config.EMAIL_USER or not Config.EMAIL_PASSWORD:
print("❌ Email credentials not configured")
print(" Set EMAIL_USER and EMAIL_PASSWORD in .env file")
return {
'success': False,
'error': 'Email credentials not configured'
}
# Get articles
max_articles = max_articles or Config.MAX_ARTICLES
print(f"\nFetching latest {max_articles} articles with AI summaries...")
articles = get_latest_articles(max_articles)
if not articles:
print("❌ No articles with summaries found")
print(" Run the crawler with Ollama enabled first")
return {
'success': False,
'error': 'No articles with summaries'
}
print(f"✓ Found {len(articles)} articles")
# Get subscribers
if test_email:
subscribers = [test_email]
print(f"\n🧪 Test mode: Sending to {test_email} only")
else:
print("\nFetching active subscribers...")
subscribers = get_active_subscribers()
print(f"✓ Found {len(subscribers)} active subscriber(s)")
if not subscribers:
print("❌ No active subscribers found")
return {
'success': False,
'error': 'No active subscribers'
}
# Render newsletter
print("\nRendering newsletter HTML...")
html_content = render_newsletter_html(articles)
print("✓ Newsletter rendered")
# Send to subscribers
subject = f"Munich News Daily - {datetime.now().strftime('%B %d, %Y')}"
print(f"\nSending newsletter: '{subject}'")
print("-" * 70)
sent_count = 0
failed_count = 0
errors = []
for i, email in enumerate(subscribers, 1):
print(f"[{i}/{len(subscribers)}] Sending to {email}...", end=' ')
success, error = send_email(email, subject, html_content)
if success:
print("✓")
sent_count += 1
else:
print(f"✗ {error}")
failed_count += 1
errors.append({'email': email, 'error': error})
# Summary
print("\n" + "="*70)
print("📊 Sending Complete")
print("="*70)
print(f"✓ Successfully sent: {sent_count}")
print(f"✗ Failed: {failed_count}")
print(f"📰 Articles included: {len(articles)}")
print("="*70 + "\n")
return {
'success': True,
'sent_count': sent_count,
'failed_count': failed_count,
'total_subscribers': len(subscribers),
'article_count': len(articles),
'errors': errors
}
def preview_newsletter(max_articles=None):
"""
Generate newsletter HTML for preview (doesn't send)
Args:
max_articles: Maximum number of articles to include
Returns:
str: HTML content
"""
max_articles = max_articles or Config.MAX_ARTICLES
articles = get_latest_articles(max_articles)
if not articles:
return "<h1>No articles with summaries found</h1><p>Run the crawler with Ollama enabled first.</p>"
return render_newsletter_html(articles)
if __name__ == '__main__':
import sys
# Parse command line arguments
if len(sys.argv) > 1:
command = sys.argv[1]
if command == 'preview':
# Generate preview HTML
html = preview_newsletter()
output_file = 'newsletter_preview.html'
with open(output_file, 'w', encoding='utf-8') as f:
f.write(html)
print(f"✓ Preview saved to {output_file}")
print(f" Open it in your browser to see the newsletter")
elif command == 'test':
# Send test email
if len(sys.argv) < 3:
print("Usage: python sender_service.py test <email>")
sys.exit(1)
test_email = sys.argv[2]
send_newsletter(test_email=test_email)
elif command == 'send':
# Send to all subscribers
max_articles = int(sys.argv[2]) if len(sys.argv) > 2 else None
send_newsletter(max_articles=max_articles)
else:
print("Unknown command. Usage:")
print(" python sender_service.py preview - Generate HTML preview")
print(" python sender_service.py test <email> - Send test email")
print(" python sender_service.py send [count] - Send to all subscribers")
else:
# Default: send newsletter
send_newsletter()