update
This commit is contained in:
299
docs/REFERENCE.md
Normal file
299
docs/REFERENCE.md
Normal file
@@ -0,0 +1,299 @@
|
||||
# Quick Reference
|
||||
|
||||
Essential commands and information.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Start services
|
||||
./start-with-gpu.sh # With GPU auto-detection
|
||||
docker-compose up -d # Without GPU
|
||||
|
||||
# Stop services
|
||||
docker-compose down
|
||||
|
||||
# View logs
|
||||
docker-compose logs -f
|
||||
docker-compose logs -f crawler # Specific service
|
||||
|
||||
# Restart
|
||||
docker-compose restart
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Docker Commands
|
||||
|
||||
```bash
|
||||
# Service status
|
||||
docker-compose ps
|
||||
|
||||
# Rebuild
|
||||
docker-compose up -d --build
|
||||
|
||||
# Execute command in container
|
||||
docker-compose exec backend python -c "print('hello')"
|
||||
docker-compose exec crawler python crawler_service.py 2
|
||||
|
||||
# View container logs
|
||||
docker logs munich-news-backend
|
||||
docker logs munich-news-crawler
|
||||
|
||||
# Resource usage
|
||||
docker stats
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Commands
|
||||
|
||||
```bash
|
||||
# Health check
|
||||
curl http://localhost:5001/health
|
||||
|
||||
# System stats
|
||||
curl http://localhost:5001/api/admin/stats
|
||||
|
||||
# Trigger crawl
|
||||
curl -X POST http://localhost:5001/api/admin/trigger-crawl \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"max_articles": 5}'
|
||||
|
||||
# Send test email
|
||||
curl -X POST http://localhost:5001/api/admin/send-test-email \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"email": "test@example.com"}'
|
||||
|
||||
# Send newsletter to all
|
||||
curl -X POST http://localhost:5001/api/admin/send-newsletter \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"max_articles": 10}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## GPU Commands
|
||||
|
||||
```bash
|
||||
# Check GPU availability
|
||||
./check-gpu.sh
|
||||
|
||||
# Start with GPU
|
||||
./start-with-gpu.sh
|
||||
docker-compose -f docker-compose.yml -f docker-compose.gpu.yml up -d
|
||||
|
||||
# Check GPU usage
|
||||
docker exec munich-news-ollama nvidia-smi
|
||||
|
||||
# Monitor GPU
|
||||
watch -n 1 'docker exec munich-news-ollama nvidia-smi'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MongoDB Commands
|
||||
|
||||
```bash
|
||||
# Access MongoDB shell
|
||||
docker-compose exec mongodb mongosh munich_news -u admin -p changeme --authenticationDatabase admin
|
||||
|
||||
# Count documents
|
||||
db.articles.countDocuments({})
|
||||
db.subscribers.countDocuments({status: 'active'})
|
||||
|
||||
# Find articles
|
||||
db.articles.find().limit(5).pretty()
|
||||
|
||||
# Clear articles
|
||||
db.articles.deleteMany({})
|
||||
|
||||
# Add subscriber
|
||||
db.subscribers.insertOne({
|
||||
email: "user@example.com",
|
||||
subscribed_at: new Date(),
|
||||
status: "active"
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Commands
|
||||
|
||||
```bash
|
||||
# Test Ollama setup
|
||||
./test-ollama-setup.sh
|
||||
|
||||
# Test MongoDB connectivity
|
||||
./test-mongodb-connectivity.sh
|
||||
|
||||
# Test newsletter API
|
||||
./test-newsletter-api.sh
|
||||
|
||||
# Test crawl (2 articles)
|
||||
docker-compose exec crawler python crawler_service.py 2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### Single .env File
|
||||
|
||||
Location: `backend/.env`
|
||||
|
||||
**Key Settings:**
|
||||
```env
|
||||
# MongoDB (Docker service name)
|
||||
MONGODB_URI=mongodb://admin:changeme@mongodb:27017/
|
||||
|
||||
# Email
|
||||
SMTP_SERVER=smtp.gmail.com
|
||||
SMTP_PORT=587
|
||||
EMAIL_USER=your@email.com
|
||||
EMAIL_PASSWORD=your-password
|
||||
|
||||
# Ollama (Internal Docker network)
|
||||
OLLAMA_ENABLED=true
|
||||
OLLAMA_BASE_URL=http://ollama:11434
|
||||
OLLAMA_MODEL=phi3:latest
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Port Exposure
|
||||
|
||||
| Service | Port | Exposed | Access |
|
||||
|---------|------|---------|--------|
|
||||
| Backend | 5001 | ✅ Yes | Host, External |
|
||||
| MongoDB | 27017 | ❌ No | Internal only |
|
||||
| Ollama | 11434 | ❌ No | Internal only |
|
||||
| Crawler | - | ❌ No | Internal only |
|
||||
| Sender | - | ❌ No | Internal only |
|
||||
|
||||
**Verify:**
|
||||
```bash
|
||||
docker ps --format "table {{.Names}}\t{{.Ports}}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance
|
||||
|
||||
### CPU Mode
|
||||
- Translation: ~1.5s per title
|
||||
- Summarization: ~8s per article
|
||||
- 10 Articles: ~115s
|
||||
|
||||
### GPU Mode (5-10x faster)
|
||||
- Translation: ~0.3s per title
|
||||
- Summarization: ~2s per article
|
||||
- 10 Articles: ~31s
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Service Won't Start
|
||||
```bash
|
||||
docker-compose logs <service-name>
|
||||
docker-compose restart <service-name>
|
||||
docker-compose up -d --build <service-name>
|
||||
```
|
||||
|
||||
### MongoDB Connection Issues
|
||||
```bash
|
||||
# Check service
|
||||
docker-compose ps mongodb
|
||||
|
||||
# Test connection
|
||||
docker-compose exec backend python -c "from database import articles_collection; print(articles_collection.count_documents({}))"
|
||||
```
|
||||
|
||||
### Ollama Issues
|
||||
```bash
|
||||
# Check model
|
||||
docker-compose exec ollama ollama list
|
||||
|
||||
# Pull model manually
|
||||
docker-compose exec ollama ollama pull phi3:latest
|
||||
|
||||
# Check logs
|
||||
docker-compose logs ollama
|
||||
```
|
||||
|
||||
### GPU Not Working
|
||||
```bash
|
||||
# Check GPU
|
||||
nvidia-smi
|
||||
|
||||
# Check Docker GPU access
|
||||
docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu22.04 nvidia-smi
|
||||
|
||||
# Check Ollama GPU
|
||||
docker exec munich-news-ollama nvidia-smi
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Recent Updates (November 2025)
|
||||
|
||||
### New Features
|
||||
- ✅ GPU acceleration (5-10x faster)
|
||||
- ✅ Integrated Ollama service
|
||||
- ✅ Send newsletter to all subscribers API
|
||||
- ✅ Article title translation (German → English)
|
||||
- ✅ Enhanced security (network isolation)
|
||||
|
||||
### Security Improvements
|
||||
- MongoDB internal-only (not exposed)
|
||||
- Ollama internal-only (not exposed)
|
||||
- Only Backend API exposed (port 5001)
|
||||
- 66% reduction in attack surface
|
||||
|
||||
### Configuration Changes
|
||||
- MongoDB URI uses `mongodb` (not `localhost`)
|
||||
- Ollama URL uses `http://ollama:11434`
|
||||
- Single `.env` file in `backend/`
|
||||
|
||||
### New Scripts
|
||||
- `start-with-gpu.sh` - Auto-detect GPU and start
|
||||
- `check-gpu.sh` - Check GPU availability
|
||||
- `test-ollama-setup.sh` - Test Ollama
|
||||
- `test-mongodb-connectivity.sh` - Test MongoDB
|
||||
- `test-newsletter-api.sh` - Test newsletter API
|
||||
|
||||
---
|
||||
|
||||
## Changelog
|
||||
|
||||
### November 2025
|
||||
- Added GPU acceleration support
|
||||
- Integrated Ollama into Docker Compose
|
||||
- Added newsletter API endpoint
|
||||
- Improved network security
|
||||
- Added article title translation
|
||||
- Consolidated documentation
|
||||
- Added helper scripts
|
||||
|
||||
### Key Changes
|
||||
- Ollama now runs in Docker (no external server needed)
|
||||
- MongoDB and Ollama are internal-only
|
||||
- GPU support with automatic detection
|
||||
- Subscriber status system documented
|
||||
- All docs consolidated and updated
|
||||
|
||||
---
|
||||
|
||||
## Links
|
||||
|
||||
- **Setup Guide**: [SETUP.md](SETUP.md)
|
||||
- **API Reference**: [API.md](API.md)
|
||||
- **Architecture**: [ARCHITECTURE.md](ARCHITECTURE.md)
|
||||
- **Security**: [SECURITY.md](SECURITY.md)
|
||||
- **Features**: [FEATURES.md](FEATURES.md)
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** November 2025
|
||||
Reference in New Issue
Block a user