dongho/Munich-news
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Dongho Kim
2025-11-11 17:40:29 +01:00
parent 901e8166cd
commit 75a6973a49
11 changed files with 1028 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files

205
NEWSLETTER_API_UPDATE.md Normal file
View File

@@ -0,0 +1,205 @@
# Newsletter API Update
## Summary
Added a new API endpoint to send newsletters to all active subscribers instead of requiring a specific email address.
## New Endpoint
### Send Newsletter to All Subscribers
```http
POST /api/admin/send-newsletter
```
**Request Body** (optional):
```json
{
"max_articles": 10
}
```
**Response**:
```json
{
"success": true,
"message": "Newsletter sent successfully to 45 subscribers",
"subscriber_count": 45,
"max_articles": 10,
"output": "... sender output ...",
"errors": ""
}
```
## Usage Examples
### Send Newsletter to All Subscribers
```bash
# Send with default settings (10 articles)
curl -X POST http://localhost:5001/api/admin/send-newsletter \
-H "Content-Type: application/json"
# Send with custom article count
curl -X POST http://localhost:5001/api/admin/send-newsletter \
-H "Content-Type: application/json" \
-d '{"max_articles": 15}'
```
### Complete Workflow
```bash
# 1. Check subscriber count
curl http://localhost:5001/api/admin/stats | jq '.subscribers'
# 2. Crawl fresh articles
curl -X POST http://localhost:5001/api/admin/trigger-crawl \
-H "Content-Type: application/json" \
-d '{"max_articles": 10}'
# 3. Wait for crawl to complete
sleep 60
# 4. Send newsletter to all active subscribers
curl -X POST http://localhost:5001/api/admin/send-newsletter \
-H "Content-Type: application/json" \
-d '{"max_articles": 10}'
```
## Comparison with Test Email
### Send Test Email (Existing)
- Sends to **one specific email address**
- Useful for testing newsletter content
- No tracking recorded in database
- Fast (single email)
```bash
curl -X POST http://localhost:5001/api/admin/send-test-email \
-H "Content-Type: application/json" \
-d '{"email": "test@example.com"}'
```
### Send Newsletter (New)
- Sends to **all active subscribers**
- Production newsletter sending
- Full tracking (opens, clicks)
- May take time for large lists
```bash
curl -X POST http://localhost:5001/api/admin/send-newsletter \
-H "Content-Type: application/json"
```
## Features
### Subscriber Filtering
- Only sends to subscribers with `status: 'active'`
- Skips inactive, unsubscribed, or bounced subscribers
- Returns error if no active subscribers found
### Tracking
- Includes tracking pixel for open tracking
- Includes click tracking for all article links
- Records send time and newsletter ID
- Stores in `newsletter_sends` collection
### Error Handling
- Validates subscriber count before sending
- Returns detailed error messages
- Includes sender output and errors in response
- 5-minute timeout for large lists
## Testing
### Interactive Test Script
```bash
./test-newsletter-api.sh
```
This script will:
1. Show current subscriber stats
2. Optionally send test email to your address
3. Optionally send newsletter to all subscribers
### Manual Testing
```bash
# 1. Check subscribers
curl http://localhost:5001/api/admin/stats
# 2. Send newsletter
curl -X POST http://localhost:5001/api/admin/send-newsletter \
-H "Content-Type: application/json" \
-d '{"max_articles": 2}'
# 3. Check results
curl http://localhost:5001/api/admin/stats
```
## Security Considerations
⚠️ **Important**: This endpoint sends emails to real subscribers!
### Recommendations
1. **Add Authentication**
```python
@require_api_key
def send_newsletter():
# ...
```
2. **Rate Limiting**
- Prevent accidental multiple sends
- Limit to once per hour/day
3. **Confirmation Required**
- Add confirmation step in UI
- Log all newsletter sends
4. **Dry Run Mode**
```json
{
"max_articles": 10,
"dry_run": true // Preview without sending
}
```
5. **Audit Logging**
- Log who triggered the send
- Log timestamp and parameters
- Track success/failure
## Files Modified
- ✅ `backend/routes/admin_routes.py` - Added new endpoint
- ✅ `docs/ADMIN_API.md` - Updated documentation
- ✅ `test-newsletter-api.sh` - Created test script
## API Endpoints Summary
| Endpoint | Purpose | Recipient |
|----------|---------|-----------|
| `/api/admin/send-test-email` | Test newsletter | Single email (specified) |
| `/api/admin/send-newsletter` | Production send | All active subscribers |
| `/api/admin/trigger-crawl` | Fetch articles | N/A |
| `/api/admin/stats` | System stats | N/A |
## Next Steps
1. **Test the endpoint:**
```bash
./test-newsletter-api.sh
```
2. **Add authentication** (recommended for production)
3. **Set up monitoring** for newsletter sends
4. **Create UI** for easier newsletter management
## Documentation
See [docs/ADMIN_API.md](docs/ADMIN_API.md) for complete API documentation.

4
QUICK_START_GPU.md
View File

@@ -56,8 +56,8 @@ docker-compose exec crawler python crawler_service.py 2
# Check translation timing
docker-compose logs crawler | grep "Title translated"
# Test Ollama API directly
curl http://localhost:11434/api/generate -d '{
# Test Ollama API (internal network only)
docker-compose exec crawler curl -s http://ollama:11434/api/generate -d '{
"model": "phi3:latest",
"prompt": "Translate to English: Guten Morgen",
"stream": false

125
SECURITY_UPDATE.md Normal file
View File

@@ -0,0 +1,125 @@
# Security Update: Ollama Internal-Only Configuration
## Summary
Ollama service has been configured to be **internal-only** and is no longer exposed to the host machine. This improves security by reducing the attack surface.
## Changes Made
### Before (Exposed)
```yaml
ollama:
ports:
- "11434:11434" # ❌ Accessible from host and external network
```
### After (Internal Only)
```yaml
ollama:
# No ports section - internal only ✓
# Only accessible within Docker network
```
## Verification
### ✓ Port Not Accessible from Host
```bash
$ nc -z -w 2 localhost 11434
# Connection refused (as expected)
```
### ✓ Accessible from Docker Services
```bash
$ docker-compose exec crawler python -c "import requests; requests.get('http://ollama:11434/api/tags')"
# ✓ Works perfectly
```
## Security Benefits
1. **No External Access**: Ollama API cannot be accessed from outside Docker network
2. **Reduced Attack Surface**: Service is not exposed to potential external threats
3. **Network Isolation**: Only authorized Docker Compose services can communicate with Ollama
4. **No Port Conflicts**: Port 11434 is not bound to host machine
## Impact on Usage
### No Change for Normal Operations ✓
- Crawler service works normally
- Translation and summarization work as before
- All Docker Compose services can access Ollama
### Testing from Host Machine
Since Ollama is internal-only, you must test from inside the Docker network:
```bash
# ✓ Test from inside a container
docker-compose exec crawler python crawler_service.py 1
# ✓ Check Ollama status
docker-compose exec crawler python -c "import requests; print(requests.get('http://ollama:11434/api/tags').json())"
# ✓ Check logs
docker-compose logs ollama
```
### If You Need External Access (Development Only)
For development/debugging, you can temporarily expose Ollama:
**Option 1: SSH Port Forward**
```bash
# Forward port through SSH (if accessing remote server)
ssh -L 11434:localhost:11434 user@server
```
**Option 2: Temporary Docker Exec**
```bash
# Run commands from inside network
docker-compose exec crawler curl http://ollama:11434/api/tags
```
**Option 3: Modify docker-compose.yml (Not Recommended)**
```yaml
ollama:
ports:
- "127.0.0.1:11434:11434" # Only localhost, not all interfaces
```
## Documentation Updated
- ✓ docker-compose.yml - Removed port exposure
- ✓ docs/OLLAMA_SETUP.md - Updated testing instructions
- ✓ docs/SECURITY_NOTES.md - Added security documentation
- ✓ test-ollama-setup.sh - Updated to test from inside network
- ✓ QUICK_START_GPU.md - Updated API testing examples
## Testing
All functionality has been verified:
- ✓ Ollama not accessible from host
- ✓ Ollama accessible from crawler service
- ✓ Translation works correctly
- ✓ Summarization works correctly
- ✓ All tests pass
## Rollback (If Needed)
If you need to expose Ollama again:
```yaml
# In docker-compose.yml
ollama:
ports:
- "11434:11434" # or "127.0.0.1:11434:11434" for localhost only
```
Then restart:
```bash
docker-compose up -d ollama
```
## Recommendation
**Keep Ollama internal-only** for production deployments. This is the most secure configuration and sufficient for normal operations.
Only expose Ollama if you have a specific need for external access, and always bind to `127.0.0.1` (localhost only), never `0.0.0.0` (all interfaces).

73
backend/routes/admin_routes.py
View File

@@ -143,6 +143,77 @@ def send_test_email():
}), 500
@admin_bp.route('/api/admin/send-newsletter', methods=['POST'])
def send_newsletter():
"""
Send newsletter to all active subscribers
Request body (optional):
{
"max_articles": 10 // Optional, defaults to 10
}
"""
try:
data = request.get_json() or {}
max_articles = data.get('max_articles', 10)
# Validate max_articles
if not isinstance(max_articles, int) or max_articles < 1 or max_articles > 50:
return jsonify({
'success': False,
'error': 'max_articles must be an integer between 1 and 50'
}), 400
# Get subscriber count first
from database import subscribers_collection
subscriber_count = subscribers_collection.count_documents({'status': 'active'})
if subscriber_count == 0:
return jsonify({
'success': False,
'error': 'No active subscribers found',
'subscriber_count': 0
}), 400
# Execute sender in sender container using docker exec
try:
result = subprocess.run(
['docker', 'exec', 'munich-news-sender', 'python', 'sender_service.py', 'send', str(max_articles)],
capture_output=True,
text=True,
timeout=300 # 5 minute timeout for multiple emails
)
# Check if successful
success = result.returncode == 0
return jsonify({
'success': success,
'message': f'Newsletter {"sent successfully" if success else "failed"} to {subscriber_count} subscribers',
'subscriber_count': subscriber_count,
'max_articles': max_articles,
'output': result.stdout[-1000:] if result.stdout else '', # Last 1000 chars
'errors': result.stderr[-500:] if result.stderr else ''
}), 200 if success else 500
except FileNotFoundError:
return jsonify({
'success': False,
'error': 'Docker command not found. Make sure Docker is installed and the socket is mounted.'
}), 500
except subprocess.TimeoutExpired:
return jsonify({
'success': False,
'error': 'Newsletter sending timed out after 5 minutes'
}), 500
except Exception as e:
return jsonify({
'success': False,
'error': f'Failed to send newsletter: {str(e)}'
}), 500
@admin_bp.route('/api/admin/stats', methods=['GET'])
def get_stats():
"""Get system statistics"""
@@ -167,7 +238,7 @@ def get_stats():
},
'subscribers': {
'total': subscribers_collection.count_documents({}),
'active': subscribers_collection.count_documents({'active': True})
'active': subscribers_collection.count_documents({'status': 'active'})
},
'rss_feeds': {
'total': rss_feeds_collection.count_documents({}),

9
docker-compose.yml
View File

@@ -6,16 +6,19 @@
# 2. Start with GPU: ./start-with-gpu.sh
# Or manually: docker-compose -f docker-compose.yml -f docker-compose.gpu.yml up -d
#
# Security:
# Ollama service is internal-only (no ports exposed to host)
# Only accessible by other Docker Compose services
#
# See docs/OLLAMA_SETUP.md for detailed setup instructions
services:
# Ollama AI Service
# Ollama AI Service (Internal only - not exposed to host)
ollama:
image: ollama/ollama:latest
container_name: munich-news-ollama
restart: unless-stopped
ports:
- "11434:11434"
# No ports exposed - only accessible within Docker network
volumes:
- ollama_data:/root/.ollama
networks:

94
docs/ADMIN_API.md
View File

@@ -100,6 +100,56 @@ curl -X POST http://localhost:5001/api/admin/send-test-email \
---
### Send Newsletter to All Subscribers
Send newsletter to all active subscribers in the database.
```http
POST /api/admin/send-newsletter
```
**Request Body** (optional):
```json
{
"max_articles": 10
}
```
**Parameters**:
- `max_articles` (integer, optional): Number of articles to include (1-50, default: 10)
**Response**:
```json
{
"success": true,
"message": "Newsletter sent successfully to 45 subscribers",
"subscriber_count": 45,
"max_articles": 10,
"output": "... sender output ...",
"errors": ""
}
```
**Example**:
```bash
# Send newsletter to all subscribers
curl -X POST http://localhost:5001/api/admin/send-newsletter \
-H "Content-Type: application/json"
# Send with custom article count
curl -X POST http://localhost:5001/api/admin/send-newsletter \
-H "Content-Type: application/json" \
-d '{"max_articles": 15}'
```
**Notes**:
- Only sends to subscribers with `status: 'active'`
- Returns error if no active subscribers found
- Includes tracking pixels and click tracking
- May take several minutes for large subscriber lists
---
### Get System Statistics
Get overview statistics of the system.
@@ -156,12 +206,32 @@ curl -X POST http://localhost:5001/api/admin/trigger-crawl \
sleep 30
curl http://localhost:5001/api/admin/stats
# 4. Send test email
# 4. Send test email to yourself
curl -X POST http://localhost:5001/api/admin/send-test-email \
-H "Content-Type: application/json" \
-d '{"email": "your-email@example.com"}'
```
### Send Newsletter to All Subscribers
```bash
# 1. Check subscriber count
curl http://localhost:5001/api/admin/stats | jq '.subscribers'
# 2. Crawl fresh articles
curl -X POST http://localhost:5001/api/admin/trigger-crawl \
-H "Content-Type: application/json" \
-d '{"max_articles": 10}'
# 3. Wait for crawl to complete
sleep 60
# 4. Send newsletter to all active subscribers
curl -X POST http://localhost:5001/api/admin/send-newsletter \
-H "Content-Type: application/json" \
-d '{"max_articles": 10}'
```
### Quick Test Newsletter
```bash
@@ -180,6 +250,28 @@ curl -X POST http://localhost:5001/api/admin/trigger-crawl \
-d '{"max_articles": 20}'
```
### Daily Newsletter Workflow
```bash
# Complete daily workflow (can be automated with cron)
# 1. Crawl today's articles
curl -X POST http://localhost:5001/api/admin/trigger-crawl \
-H "Content-Type: application/json" \
-d '{"max_articles": 15}'
# 2. Wait for crawl and AI processing
sleep 120
# 3. Send to all subscribers
curl -X POST http://localhost:5001/api/admin/send-newsletter \
-H "Content-Type: application/json" \
-d '{"max_articles": 10}'
# 4. Check results
curl http://localhost:5001/api/admin/stats
```
---
## Error Responses

11
docs/OLLAMA_SETUP.md
View File

@@ -7,10 +7,11 @@ This project includes an integrated Ollama service for AI-powered summarization
## Docker Compose Setup (Recommended)
The docker-compose.yml includes an Ollama service that automatically:
- Runs Ollama server on port 11434
- Runs Ollama server (internal only, not exposed to host)
- Pulls the phi3:latest model on first startup
- Persists model data in a Docker volume
- Supports GPU acceleration (NVIDIA GPUs)
- Only accessible by other Docker Compose services for security
### GPU Support
@@ -90,8 +91,8 @@ docker-compose logs -f ollama
# Check model setup logs
docker-compose logs ollama-setup
# Verify Ollama is running
curl http://localhost:11434/api/tags
# Verify Ollama is running (from inside a container)
docker-compose exec crawler curl http://ollama:11434/api/tags
```
### First Time Setup
@@ -187,8 +188,8 @@ If you prefer to run Ollama directly on your host machine:
### Basic API Test
```bash
# Test Ollama API directly
curl http://localhost:11434/api/generate -d '{
# Test Ollama API from inside a container
docker-compose exec crawler curl -s http://ollama:11434/api/generate -d '{
"model": "phi3:latest",
"prompt": "Translate to English: Guten Morgen",
"stream": false

160
docs/SECURITY_NOTES.md Normal file
View File

@@ -0,0 +1,160 @@
# Security Notes
## Ollama Service Security
### Internal-Only Access
The Ollama service is configured to be **internal-only** and is not exposed to the host machine or external network. This provides several security benefits:
**Configuration:**
```yaml
# Ollama service has NO ports exposed
ollama:
image: ollama/ollama:latest
# No ports section - internal only
networks:
- munich-news-network
```
**Benefits:**
1. **No External Access**: Ollama API cannot be accessed from outside Docker network
2. **Reduced Attack Surface**: Service is not exposed to potential external threats
3. **Network Isolation**: Only authorized Docker Compose services can communicate with Ollama
4. **No Port Conflicts**: Port 11434 is not bound to host machine
### Accessing Ollama
**From Docker Compose Services (✓ Allowed):**
```bash
# Services use internal Docker network
OLLAMA_BASE_URL=http://ollama:11434
```
**From Host Machine (✗ Not Allowed):**
```bash
# This will NOT work - port not exposed
curl http://localhost:11434/api/tags
# Connection refused
```
**From Inside Containers (✓ Allowed):**
```bash
# Access from another container
docker-compose exec crawler curl http://ollama:11434/api/tags
```
### Why This Matters
**Security Risks of Exposed Ollama:**
- Unauthorized access to AI models
- Potential for abuse (resource consumption)
- Information disclosure through prompts
- No authentication by default
- Could be used for unintended purposes
**With Internal-Only Configuration:**
- Only your trusted services can access Ollama
- No external attack vector
- Controlled usage within your application
- Better resource management
### Testing Ollama
Since Ollama is internal-only, you must test from inside the Docker network:
```bash
# ✓ Correct way - from inside a container
docker-compose exec crawler curl -s http://ollama:11434/api/tags
# ✓ Test translation
docker-compose exec crawler python crawler_service.py 1
# ✓ Check logs
docker-compose logs ollama
```
### If You Need External Access
If you have a specific need to access Ollama from the host machine (e.g., development, debugging), you can temporarily expose it:
**Option 1: Temporary Port Forward**
```bash
# Forward port temporarily (stops when you press Ctrl+C)
docker exec -it munich-news-ollama socat TCP-LISTEN:11434,fork TCP:localhost:11434 &
```
**Option 2: Add Ports to docker-compose.yml (Not Recommended)**
```yaml
ollama:
ports:
- "127.0.0.1:11434:11434" # Only bind to localhost, not 0.0.0.0
```
**⚠️ Warning:** Only expose Ollama if absolutely necessary, and always bind to `127.0.0.1` (localhost only), never `0.0.0.0` (all interfaces).
### Other Security Considerations
**MongoDB:**
- Exposed on port 27017 for development
- Uses authentication (username/password)
- Consider restricting to localhost in production: `127.0.0.1:27017:27017`
**Backend API:**
- Exposed on port 5001 for tracking and admin functions
- Should be behind reverse proxy in production
- Consider adding authentication for admin endpoints
**Email Credentials:**
- Stored in `.env` file
- Never commit `.env` to version control
- Use environment variables in production
### Production Recommendations
1. **Use Docker Secrets** for sensitive data:
```yaml
secrets:
mongo_password:
external: true
```
2. **Restrict Network Access**:
```yaml
ports:
- "127.0.0.1:27017:27017" # MongoDB
- "127.0.0.1:5001:5001" # Backend
```
3. **Use Reverse Proxy** (nginx, Traefik):
- SSL/TLS termination
- Rate limiting
- Authentication
- Access logs
4. **Regular Updates**:
```bash
docker-compose pull
docker-compose up -d
```
5. **Monitor Logs**:
```bash
docker-compose logs -f
```
### Security Checklist
- [x] Ollama is internal-only (no exposed ports)
- [x] MongoDB uses authentication
- [x] `.env` file is in `.gitignore`
- [ ] Backend API has authentication (if needed)
- [ ] Using HTTPS in production
- [ ] Regular security updates
- [ ] Monitoring and logging enabled
- [ ] Backup strategy in place
## Reporting Security Issues
If you discover a security vulnerability, please email security@example.com (replace with your contact).
Do not open public issues for security vulnerabilities.

290
docs/SUBSCRIBER_STATUS.md Normal file
View File

@@ -0,0 +1,290 @@
# Subscriber Status System
## Overview
The newsletter system tracks subscribers with a `status` field that determines whether they receive newsletters.
## Status Field
### Database Schema
```javascript
{
_id: ObjectId("..."),
email: "user@example.com",
subscribed_at: ISODate("2025-11-11T15:50:29.478Z"),
status: "active" // or "inactive"
}
```
### Status Values
| Status | Description | Receives Newsletters |
|--------|-------------|---------------------|
| `active` | Subscribed and active | ✅ Yes |
| `inactive` | Unsubscribed | ❌ No |
## How It Works
### Subscription Flow
```
User subscribes
↓
POST /api/subscribe
↓
Create subscriber with status: 'active'
↓
User receives newsletters
```
### Unsubscription Flow
```
User unsubscribes
↓
POST /api/unsubscribe
↓
Update subscriber status: 'inactive'
↓
User stops receiving newsletters
```
### Re-subscription Flow
```
Previously unsubscribed user subscribes again
↓
POST /api/subscribe
↓
Update status: 'active' + new subscribed_at date
↓
User receives newsletters again
```
## API Endpoints
### Subscribe
```bash
curl -X POST http://localhost:5001/api/subscribe \
-H "Content-Type: application/json" \
-d '{"email": "user@example.com"}'
```
**Creates subscriber with:**
- `email`: user@example.com
- `status`: "active"
- `subscribed_at`: current timestamp
### Unsubscribe
```bash
curl -X POST http://localhost:5001/api/unsubscribe \
-H "Content-Type: application/json" \
-d '{"email": "user@example.com"}'
```
**Updates subscriber:**
- `status`: "inactive"
## Newsletter Sending
### Who Receives Newsletters
Only subscribers with `status: 'active'` receive newsletters.
**Sender Service Query:**
```python
subscribers_collection.find({'status': 'active'})
```
**Admin API Query:**
```python
subscribers_collection.count_documents({'status': 'active'})
```
### Testing
```bash
# Check active subscriber count
curl http://localhost:5001/api/admin/stats | jq '.subscribers'
# Output:
# {
# "total": 10,
# "active": 8
# }
```
## Database Operations
### Add Active Subscriber
```javascript
db.subscribers.insertOne({
email: "user@example.com",
subscribed_at: new Date(),
status: "active"
})
```
### Deactivate Subscriber
```javascript
db.subscribers.updateOne(
{ email: "user@example.com" },
{ $set: { status: "inactive" } }
)
```
### Reactivate Subscriber
```javascript
db.subscribers.updateOne(
{ email: "user@example.com" },
{ $set: {
status: "active",
subscribed_at: new Date()
}}
)
```
### Query Active Subscribers
```javascript
db.subscribers.find({ status: "active" })
```
### Count Active Subscribers
```javascript
db.subscribers.countDocuments({ status: "active" })
```
## Common Issues
### Issue: Stats show 0 active subscribers but subscribers exist
**Cause:** Old bug where stats checked `{active: true}` instead of `{status: 'active'}`
**Solution:** Fixed in latest version. Stats now correctly query `{status: 'active'}`
**Verify:**
```bash
# Check database directly
docker-compose exec mongodb mongosh munich_news -u admin -p changeme \
--authenticationDatabase admin \
--eval "db.subscribers.find({status: 'active'}).count()"
# Check via API
curl http://localhost:5001/api/admin/stats | jq '.subscribers.active'
```
### Issue: Newsletter not sending to subscribers
**Possible causes:**
1. Subscribers have `status: 'inactive'`
2. No subscribers in database
3. Email configuration issue
**Debug:**
```bash
# Check subscriber status
docker-compose exec mongodb mongosh munich_news -u admin -p changeme \
--authenticationDatabase admin \
--eval "db.subscribers.find().pretty()"
# Check active count
curl http://localhost:5001/api/admin/stats | jq '.subscribers'
# Try sending
curl -X POST http://localhost:5001/api/admin/send-newsletter \
-H "Content-Type: application/json"
```
## Migration Notes
### If you have old subscribers without status field
Run this migration:
```javascript
// Set all subscribers without status to 'active'
db.subscribers.updateMany(
{ status: { $exists: false } },
{ $set: { status: "active" } }
)
```
### If you have subscribers with `active: true/false` field
Run this migration:
```javascript
// Convert old 'active' field to 'status' field
db.subscribers.updateMany(
{ active: true },
{ $set: { status: "active" }, $unset: { active: "" } }
)
db.subscribers.updateMany(
{ active: false },
{ $set: { status: "inactive" }, $unset: { active: "" } }
)
```
## Best Practices
### 1. Always Check Status
When querying subscribers for sending:
```python
# ✅ Correct
subscribers_collection.find({'status': 'active'})
# ❌ Wrong
subscribers_collection.find({}) # Includes inactive
```
### 2. Soft Delete
Never delete subscribers - just set status to 'inactive':
```python
# ✅ Correct - preserves history
subscribers_collection.update_one(
{'email': email},
{'$set': {'status': 'inactive'}}
)
# ❌ Wrong - loses data
subscribers_collection.delete_one({'email': email})
```
### 3. Track Subscription History
Consider adding fields:
```javascript
{
email: "user@example.com",
status: "active",
subscribed_at: ISODate("2025-01-01"),
unsubscribed_at: null, // Set when status changes to inactive
resubscribed_count: 0 // Increment on re-subscription
}
```
### 4. Validate Before Sending
```python
# Check subscriber count before sending
count = subscribers_collection.count_documents({'status': 'active'})
if count == 0:
return {'error': 'No active subscribers'}
```
## Related Documentation
- [ADMIN_API.md](ADMIN_API.md) - Admin API endpoints
- [DATABASE_SCHEMA.md](DATABASE_SCHEMA.md) - Complete database schema
- [NEWSLETTER_API_UPDATE.md](../NEWSLETTER_API_UPDATE.md) - Newsletter API changes

48
test-newsletter-api.sh Executable file
View File

@@ -0,0 +1,48 @@
#!/bin/bash
# Test script for newsletter API endpoints
echo "=========================================="
echo "Newsletter API Test"
echo "=========================================="
echo ""
BASE_URL="http://localhost:5001"
# Test 1: Get stats
echo "Test 1: Get system stats"
echo "GET $BASE_URL/api/admin/stats"
curl -s $BASE_URL/api/admin/stats | python3 -m json.tool | grep -A 3 "subscribers"
echo ""
# Test 2: Send test email
echo "Test 2: Send test email"
read -p "Enter your email address for test: " TEST_EMAIL
if [ -n "$TEST_EMAIL" ]; then
echo "POST $BASE_URL/api/admin/send-test-email"
curl -s -X POST $BASE_URL/api/admin/send-test-email \
-H "Content-Type: application/json" \
-d "{\"email\": \"$TEST_EMAIL\", \"max_articles\": 2}" | python3 -m json.tool
echo ""
else
echo "Skipped (no email provided)"
echo ""
fi
# Test 3: Send newsletter to all subscribers
echo "Test 3: Send newsletter to all subscribers"
read -p "Send newsletter to all active subscribers? (y/N): " CONFIRM
if [ "$CONFIRM" = "y" ] || [ "$CONFIRM" = "Y" ]; then
echo "POST $BASE_URL/api/admin/send-newsletter"
curl -s -X POST $BASE_URL/api/admin/send-newsletter \
-H "Content-Type: application/json" \
-d '{"max_articles": 5}' | python3 -m json.tool
echo ""
else
echo "Skipped"
echo ""
fi
echo "=========================================="
echo "Test Complete"
echo "=========================================="

28
test-ollama-setup.sh
View File

@@ -117,18 +117,30 @@ echo "Test 8: Ollama service status"
if docker ps | grep -q "munich-news-ollama"; then
echo "✓ Ollama container is running"
# Test Ollama API
if curl -s http://localhost:11434/api/tags &> /dev/null; then
echo "✓ Ollama API is accessible"
# Check if crawler is running (needed to test Ollama)
if docker ps | grep -q "munich-news-crawler"; then
# Test Ollama API from inside network using Python
if docker-compose exec -T crawler python -c "import requests; requests.get('http://ollama:11434/api/tags', timeout=5)" &> /dev/null; then
echo "✓ Ollama API is accessible (internal network)"
# Check if model is available
if curl -s http://localhost:11434/api/tags | grep -q "phi3"; then
echo "✓ phi3 model is available"
# Check if model is available
if docker-compose exec -T crawler python -c "import requests; r = requests.get('http://ollama:11434/api/tags'); exit(0 if 'phi3' in r.text else 1)" &> /dev/null; then
echo "✓ phi3 model is available"
else
echo "⚠ phi3 model not found (may still be downloading)"
fi
else
echo "⚠ phi3 model not found (may still be downloading)"
echo "⚠ Ollama API not responding from crawler"
fi
else
echo "⚠ Ollama API not responding"
echo "ℹ Crawler not running (needed to test internal Ollama access)"
fi
# Verify port is NOT exposed to host
if nc -z -w 2 localhost 11434 &> /dev/null; then
echo "⚠ WARNING: Ollama port is exposed to host (should be internal only)"
else
echo "✓ Ollama is internal-only (not exposed to host)"
fi
else
echo "ℹ Ollama container not running (start with: docker-compose up -d)"