update

2025-11-11 17:40:29 +01:00
parent 901e8166cd
commit 75a6973a49
11 changed files with 1028 additions and 21 deletions
--- a/docs/ADMIN_API.md
+++ b/docs/ADMIN_API.md
@@ -100,6 +100,56 @@ curl -X POST http://localhost:5001/api/admin/send-test-email \

 ---

+### Send Newsletter to All Subscribers
+
+Send newsletter to all active subscribers in the database.
+
+```http
+POST /api/admin/send-newsletter
+```
+
+**Request Body** (optional):
+```json
+{
+  "max_articles": 10
+}
+```
+
+**Parameters**:
+- `max_articles` (integer, optional): Number of articles to include (1-50, default: 10)
+
+**Response**:
+```json
+{
+  "success": true,
+  "message": "Newsletter sent successfully to 45 subscribers",
+  "subscriber_count": 45,
+  "max_articles": 10,
+  "output": "... sender output ...",
+  "errors": ""
+}
+```
+
+**Example**:
+```bash
+# Send newsletter to all subscribers
+curl -X POST http://localhost:5001/api/admin/send-newsletter \
+  -H "Content-Type: application/json"
+
+# Send with custom article count
+curl -X POST http://localhost:5001/api/admin/send-newsletter \
+  -H "Content-Type: application/json" \
+  -d '{"max_articles": 15}'
+```
+
+**Notes**:
+- Only sends to subscribers with `status: 'active'`
+- Returns error if no active subscribers found
+- Includes tracking pixels and click tracking
+- May take several minutes for large subscriber lists
+
+---
+
 ### Get System Statistics

 Get overview statistics of the system.
@@ -156,12 +206,32 @@ curl -X POST http://localhost:5001/api/admin/trigger-crawl \
 sleep 30
 curl http://localhost:5001/api/admin/stats

-# 4. Send test email
+# 4. Send test email to yourself
 curl -X POST http://localhost:5001/api/admin/send-test-email \
  -H "Content-Type: application/json" \
  -d '{"email": "your-email@example.com"}'
 ```

+### Send Newsletter to All Subscribers
+
+```bash
+# 1. Check subscriber count
+curl http://localhost:5001/api/admin/stats | jq '.subscribers'
+
+# 2. Crawl fresh articles
+curl -X POST http://localhost:5001/api/admin/trigger-crawl \
+  -H "Content-Type: application/json" \
+  -d '{"max_articles": 10}'
+
+# 3. Wait for crawl to complete
+sleep 60
+
+# 4. Send newsletter to all active subscribers
+curl -X POST http://localhost:5001/api/admin/send-newsletter \
+  -H "Content-Type: application/json" \
+  -d '{"max_articles": 10}'
+```
+
 ### Quick Test Newsletter

 ```bash
@@ -180,6 +250,28 @@ curl -X POST http://localhost:5001/api/admin/trigger-crawl \
  -d '{"max_articles": 20}'
 ```

+### Daily Newsletter Workflow
+
+```bash
+# Complete daily workflow (can be automated with cron)
+
+# 1. Crawl today's articles
+curl -X POST http://localhost:5001/api/admin/trigger-crawl \
+  -H "Content-Type: application/json" \
+  -d '{"max_articles": 15}'
+
+# 2. Wait for crawl and AI processing
+sleep 120
+
+# 3. Send to all subscribers
+curl -X POST http://localhost:5001/api/admin/send-newsletter \
+  -H "Content-Type: application/json" \
+  -d '{"max_articles": 10}'
+
+# 4. Check results
+curl http://localhost:5001/api/admin/stats
+```
+
 ---

 ## Error Responses
--- a/docs/OLLAMA_SETUP.md
+++ b/docs/OLLAMA_SETUP.md
@@ -7,10 +7,11 @@ This project includes an integrated Ollama service for AI-powered summarization
 ## Docker Compose Setup (Recommended)

 The docker-compose.yml includes an Ollama service that automatically:
- Runs Ollama server on port 11434
+- Runs Ollama server (internal only, not exposed to host)
 - Pulls the phi3:latest model on first startup
 - Persists model data in a Docker volume
 - Supports GPU acceleration (NVIDIA GPUs)
+- Only accessible by other Docker Compose services for security

 ### GPU Support

@@ -90,8 +91,8 @@ docker-compose logs -f ollama
 # Check model setup logs
 docker-compose logs ollama-setup

-# Verify Ollama is running
-curl http://localhost:11434/api/tags
+# Verify Ollama is running (from inside a container)
+docker-compose exec crawler curl http://ollama:11434/api/tags
 ```

 ### First Time Setup
@@ -187,8 +188,8 @@ If you prefer to run Ollama directly on your host machine:

 ### Basic API Test
 ```bash
-# Test Ollama API directly
-curl http://localhost:11434/api/generate -d '{
+# Test Ollama API from inside a container
+docker-compose exec crawler curl -s http://ollama:11434/api/generate -d '{
  "model": "phi3:latest",
  "prompt": "Translate to English: Guten Morgen",
  "stream": false
--- a/docs/SECURITY_NOTES.md
+++ b/docs/SECURITY_NOTES.md
@@ -0,0 +1,160 @@
+# Security Notes
+
+## Ollama Service Security
+
+### Internal-Only Access
+
+The Ollama service is configured to be **internal-only** and is not exposed to the host machine or external network. This provides several security benefits:
+
+**Configuration:**
+```yaml
+# Ollama service has NO ports exposed
+ollama:
+  image: ollama/ollama:latest
+  # No ports section - internal only
+  networks:
+    - munich-news-network
+```
+
+**Benefits:**
+1. **No External Access**: Ollama API cannot be accessed from outside Docker network
+2. **Reduced Attack Surface**: Service is not exposed to potential external threats
+3. **Network Isolation**: Only authorized Docker Compose services can communicate with Ollama
+4. **No Port Conflicts**: Port 11434 is not bound to host machine
+
+### Accessing Ollama
+
+**From Docker Compose Services (✓ Allowed):**
+```bash
+# Services use internal Docker network
+OLLAMA_BASE_URL=http://ollama:11434
+```
+
+**From Host Machine (✗ Not Allowed):**
+```bash
+# This will NOT work - port not exposed
+curl http://localhost:11434/api/tags
+# Connection refused
+```
+
+**From Inside Containers (✓ Allowed):**
+```bash
+# Access from another container
+docker-compose exec crawler curl http://ollama:11434/api/tags
+```
+
+### Why This Matters
+
+**Security Risks of Exposed Ollama:**
+- Unauthorized access to AI models
+- Potential for abuse (resource consumption)
+- Information disclosure through prompts
+- No authentication by default
+- Could be used for unintended purposes
+
+**With Internal-Only Configuration:**
+- Only your trusted services can access Ollama
+- No external attack vector
+- Controlled usage within your application
+- Better resource management
+
+### Testing Ollama
+
+Since Ollama is internal-only, you must test from inside the Docker network:
+
+```bash
+# ✓ Correct way - from inside a container
+docker-compose exec crawler curl -s http://ollama:11434/api/tags
+
+# ✓ Test translation
+docker-compose exec crawler python crawler_service.py 1
+
+# ✓ Check logs
+docker-compose logs ollama
+```
+
+### If You Need External Access
+
+If you have a specific need to access Ollama from the host machine (e.g., development, debugging), you can temporarily expose it:
+
+**Option 1: Temporary Port Forward**
+```bash
+# Forward port temporarily (stops when you press Ctrl+C)
+docker exec -it munich-news-ollama socat TCP-LISTEN:11434,fork TCP:localhost:11434 &
+```
+
+**Option 2: Add Ports to docker-compose.yml (Not Recommended)**
+```yaml
+ollama:
+  ports:
+    - "127.0.0.1:11434:11434"  # Only bind to localhost, not 0.0.0.0
+```
+
+**⚠️ Warning:** Only expose Ollama if absolutely necessary, and always bind to `127.0.0.1` (localhost only), never `0.0.0.0` (all interfaces).
+
+### Other Security Considerations
+
+**MongoDB:**
+- Exposed on port 27017 for development
+- Uses authentication (username/password)
+- Consider restricting to localhost in production: `127.0.0.1:27017:27017`
+
+**Backend API:**
+- Exposed on port 5001 for tracking and admin functions
+- Should be behind reverse proxy in production
+- Consider adding authentication for admin endpoints
+
+**Email Credentials:**
+- Stored in `.env` file
+- Never commit `.env` to version control
+- Use environment variables in production
+
+### Production Recommendations
+
+1. **Use Docker Secrets** for sensitive data:
+   ```yaml
+   secrets:
+     mongo_password:
+       external: true
+   ```
+
+2. **Restrict Network Access**:
+   ```yaml
+   ports:
+     - "127.0.0.1:27017:27017"  # MongoDB
+     - "127.0.0.1:5001:5001"    # Backend
+   ```
+
+3. **Use Reverse Proxy** (nginx, Traefik):
+   - SSL/TLS termination
+   - Rate limiting
+   - Authentication
+   - Access logs
+
+4. **Regular Updates**:
+   ```bash
+   docker-compose pull
+   docker-compose up -d
+   ```
+
+5. **Monitor Logs**:
+   ```bash
+   docker-compose logs -f
+   ```
+
+### Security Checklist
+
+- [x] Ollama is internal-only (no exposed ports)
+- [x] MongoDB uses authentication
+- [x] `.env` file is in `.gitignore`
+- [ ] Backend API has authentication (if needed)
+- [ ] Using HTTPS in production
+- [ ] Regular security updates
+- [ ] Monitoring and logging enabled
+- [ ] Backup strategy in place
+
+## Reporting Security Issues
+
+If you discover a security vulnerability, please email security@example.com (replace with your contact).
+
+Do not open public issues for security vulnerabilities.
--- a/docs/SUBSCRIBER_STATUS.md
+++ b/docs/SUBSCRIBER_STATUS.md
@@ -0,0 +1,290 @@
+# Subscriber Status System
+
+## Overview
+
+The newsletter system tracks subscribers with a `status` field that determines whether they receive newsletters.
+
+## Status Field
+
+### Database Schema
+
+```javascript
+{
+  _id: ObjectId("..."),
+  email: "user@example.com",
+  subscribed_at: ISODate("2025-11-11T15:50:29.478Z"),
+  status: "active"  // or "inactive"
+}
+```
+
+### Status Values
+
+| Status | Description | Receives Newsletters |
+|--------|-------------|---------------------|
+| `active` | Subscribed and active | ✅ Yes |
+| `inactive` | Unsubscribed | ❌ No |
+
+## How It Works
+
+### Subscription Flow
+
+```
+User subscribes
+    ↓
+POST /api/subscribe
+    ↓
+Create subscriber with status: 'active'
+    ↓
+User receives newsletters
+```
+
+### Unsubscription Flow
+
+```
+User unsubscribes
+    ↓
+POST /api/unsubscribe
+    ↓
+Update subscriber status: 'inactive'
+    ↓
+User stops receiving newsletters
+```
+
+### Re-subscription Flow
+
+```
+Previously unsubscribed user subscribes again
+    ↓
+POST /api/subscribe
+    ↓
+Update status: 'active' + new subscribed_at date
+    ↓
+User receives newsletters again
+```
+
+## API Endpoints
+
+### Subscribe
+
+```bash
+curl -X POST http://localhost:5001/api/subscribe \
+  -H "Content-Type: application/json" \
+  -d '{"email": "user@example.com"}'
+```
+
+**Creates subscriber with:**
+- `email`: user@example.com
+- `status`: "active"
+- `subscribed_at`: current timestamp
+
+### Unsubscribe
+
+```bash
+curl -X POST http://localhost:5001/api/unsubscribe \
+  -H "Content-Type: application/json" \
+  -d '{"email": "user@example.com"}'
+```
+
+**Updates subscriber:**
+- `status`: "inactive"
+
+## Newsletter Sending
+
+### Who Receives Newsletters
+
+Only subscribers with `status: 'active'` receive newsletters.
+
+**Sender Service Query:**
+```python
+subscribers_collection.find({'status': 'active'})
+```
+
+**Admin API Query:**
+```python
+subscribers_collection.count_documents({'status': 'active'})
+```
+
+### Testing
+
+```bash
+# Check active subscriber count
+curl http://localhost:5001/api/admin/stats | jq '.subscribers'
+
+# Output:
+# {
+#   "total": 10,
+#   "active": 8
+# }
+```
+
+## Database Operations
+
+### Add Active Subscriber
+
+```javascript
+db.subscribers.insertOne({
+  email: "user@example.com",
+  subscribed_at: new Date(),
+  status: "active"
+})
+```
+
+### Deactivate Subscriber
+
+```javascript
+db.subscribers.updateOne(
+  { email: "user@example.com" },
+  { $set: { status: "inactive" } }
+)
+```
+
+### Reactivate Subscriber
+
+```javascript
+db.subscribers.updateOne(
+  { email: "user@example.com" },
+  { $set: { 
+    status: "active",
+    subscribed_at: new Date()
+  }}
+)
+```
+
+### Query Active Subscribers
+
+```javascript
+db.subscribers.find({ status: "active" })
+```
+
+### Count Active Subscribers
+
+```javascript
+db.subscribers.countDocuments({ status: "active" })
+```
+
+## Common Issues
+
+### Issue: Stats show 0 active subscribers but subscribers exist
+
+**Cause:** Old bug where stats checked `{active: true}` instead of `{status: 'active'}`
+
+**Solution:** Fixed in latest version. Stats now correctly query `{status: 'active'}`
+
+**Verify:**
+```bash
+# Check database directly
+docker-compose exec mongodb mongosh munich_news -u admin -p changeme \
+  --authenticationDatabase admin \
+  --eval "db.subscribers.find({status: 'active'}).count()"
+
+# Check via API
+curl http://localhost:5001/api/admin/stats | jq '.subscribers.active'
+```
+
+### Issue: Newsletter not sending to subscribers
+
+**Possible causes:**
+1. Subscribers have `status: 'inactive'`
+2. No subscribers in database
+3. Email configuration issue
+
+**Debug:**
+```bash
+# Check subscriber status
+docker-compose exec mongodb mongosh munich_news -u admin -p changeme \
+  --authenticationDatabase admin \
+  --eval "db.subscribers.find().pretty()"
+
+# Check active count
+curl http://localhost:5001/api/admin/stats | jq '.subscribers'
+
+# Try sending
+curl -X POST http://localhost:5001/api/admin/send-newsletter \
+  -H "Content-Type: application/json"
+```
+
+## Migration Notes
+
+### If you have old subscribers without status field
+
+Run this migration:
+
+```javascript
+// Set all subscribers without status to 'active'
+db.subscribers.updateMany(
+  { status: { $exists: false } },
+  { $set: { status: "active" } }
+)
+```
+
+### If you have subscribers with `active: true/false` field
+
+Run this migration:
+
+```javascript
+// Convert old 'active' field to 'status' field
+db.subscribers.updateMany(
+  { active: true },
+  { $set: { status: "active" }, $unset: { active: "" } }
+)
+
+db.subscribers.updateMany(
+  { active: false },
+  { $set: { status: "inactive" }, $unset: { active: "" } }
+)
+```
+
+## Best Practices
+
+### 1. Always Check Status
+
+When querying subscribers for sending:
+```python
+# ✅ Correct
+subscribers_collection.find({'status': 'active'})
+
+# ❌ Wrong
+subscribers_collection.find({})  # Includes inactive
+```
+
+### 2. Soft Delete
+
+Never delete subscribers - just set status to 'inactive':
+```python
+# ✅ Correct - preserves history
+subscribers_collection.update_one(
+    {'email': email},
+    {'$set': {'status': 'inactive'}}
+)
+
+# ❌ Wrong - loses data
+subscribers_collection.delete_one({'email': email})
+```
+
+### 3. Track Subscription History
+
+Consider adding fields:
+```javascript
+{
+  email: "user@example.com",
+  status: "active",
+  subscribed_at: ISODate("2025-01-01"),
+  unsubscribed_at: null,  // Set when status changes to inactive
+  resubscribed_count: 0   // Increment on re-subscription
+}
+```
+
+### 4. Validate Before Sending
+
+```python
+# Check subscriber count before sending
+count = subscribers_collection.count_documents({'status': 'active'})
+if count == 0:
+    return {'error': 'No active subscribers'}
+```
+
+## Related Documentation
+
+- [ADMIN_API.md](ADMIN_API.md) - Admin API endpoints
+- [DATABASE_SCHEMA.md](DATABASE_SCHEMA.md) - Complete database schema
+- [NEWSLETTER_API_UPDATE.md](../NEWSLETTER_API_UPDATE.md) - Newsletter API changes