update

2025-11-11 16:58:03 +01:00
parent f23f4b71d8
commit 324751eb5d
14 changed files with 1108 additions and 18 deletions
--- a/.kiro/specs/article-title-translation/design.md
+++ b/.kiro/specs/article-title-translation/design.md
@@ -0,0 +1,328 @@
+# Design Document: Article Title Translation
+
+## Overview
+
+This feature extends the existing Ollama AI integration to translate German article titles to English during the crawling process. The translation will be performed immediately after article content extraction and before AI summarization. Both the original German title and English translation will be stored in the MongoDB article document, and the newsletter template will be updated to display the English title prominently with the original as a subtitle.
+
+The design leverages the existing Ollama infrastructure (same server, configuration, and error handling patterns) to minimize complexity and maintain consistency with the current summarization feature.
+
+## Architecture
+
+### Component Interaction Flow
+
+```
+RSS Feed Entry
+    ↓
+Crawler Service (extract_article_content)
+    ↓
+Article Data (with German title)
+    ↓
+Ollama Client (translate_title) ← New Method
+    ↓
+Translation Result
+    ↓
+Crawler Service (prepare article_doc)
+    ↓
+MongoDB (articles collection with title + title_en)
+    ↓
+Newsletter Service (fetch articles)
+    ↓
+Newsletter Template (display English title + German subtitle)
+    ↓
+Email to Subscribers
+```
+
+### Integration Points
+
+1. **Ollama Client** - Add new `translate_title()` method alongside existing `summarize_article()` method
+2. **Crawler Service** - Call translation after content extraction, before summarization
+3. **Article Document Schema** - Add `title_en` and `translated_at` fields
+4. **Newsletter Template** - Update title display logic to show English/German titles
+
+## Components and Interfaces
+
+### 1. Ollama Client Extension
+
+**New Method: `translate_title(title, target_language='English')`**
+
+```python
+def translate_title(self, title, target_language='English'):
+    """
+    Translate article title to target language
+    
+    Args:
+        title (str): Original German title
+        target_language (str): Target language (default: 'English')
+    
+    Returns:
+        dict: {
+            'success': bool,
+            'translated_title': str or None,
+            'error': str or None,
+            'duration': float
+        }
+    """
+```
+
+**Implementation Details:**
+
+- **Prompt Engineering**: Clear, concise prompt instructing the model to translate only the headline without explanations
+- **Temperature**: 0.3 (lower than summarization's 0.7) for more consistent, deterministic translations
+- **Token Limit**: 100 tokens (sufficient for title-length outputs)
+- **Response Cleaning**: 
+  - Remove surrounding quotes (single and double)
+  - Extract first line only (ignore any extra text)
+  - Trim whitespace
+- **Error Handling**: Same pattern as `summarize_article()` - catch timeouts, connection errors, HTTP errors
+- **Validation**: Check for empty title input before making API call
+
+### 2. Crawler Service Integration
+
+**Location**: In `crawl_rss_feed()` function, after content extraction
+
+**Execution Order**:
+1. Extract article content (existing)
+2. **Translate title** (new)
+3. Summarize article (existing)
+4. Save to database (modified)
+
+**Implementation Pattern**:
+
+```python
+# After article_data extraction
+translation_result = None
+original_title = article_data.get('title') or entry.get('title', '')
+
+if Config.OLLAMA_ENABLED:
+    # Translate title
+    print(f"   🌐 Translating title...")
+    translation_result = ollama_client.translate_title(original_title)
+    
+    if translation_result and translation_result['success']:
+        print(f"   ✓ Title translated ({translation_result['duration']:.1f}s)")
+    else:
+        print(f"   ⚠ Translation failed: {translation_result['error']}")
+    
+    # Then summarize (existing code)
+    ...
+```
+
+**Console Output Format**:
+- Success: `✓ Title translated (0.8s)`
+- Failure: `⚠ Translation failed: Request timed out`
+
+### 3. Data Models
+
+**MongoDB Article Document Schema Extension**:
+
+```javascript
+{
+  // Existing fields
+  title: String,              // Original German title
+  author: String,
+  link: String,
+  content: String,
+  summary: String,
+  word_count: Number,
+  summary_word_count: Number,
+  source: String,
+  category: String,
+  published_at: Date,
+  crawled_at: Date,
+  summarized_at: Date,
+  created_at: Date,
+  
+  // New fields
+  title_en: String,           // English translation of title (nullable)
+  translated_at: Date         // Timestamp when translation completed (nullable)
+}
+```
+
+**Field Behavior**:
+- `title_en`: NULL if translation fails or Ollama is disabled
+- `translated_at`: NULL if translation fails, set to `datetime.utcnow()` on success
+
+### 4. Newsletter Template Updates
+
+**Current Title Display**:
+```html
+<h2 style="...">
+    {{ article.title }}
+</h2>
+```
+
+**New Title Display Logic**:
+```html
+<!-- Primary title: English if available, otherwise German -->
+<h2 style="margin: 12px 0 8px 0; font-size: 19px; font-weight: 700; line-height: 1.3; color: #1a1a1a;">
+    {{ article.title_en if article.title_en else article.title }}
+</h2>
+
+<!-- Subtitle: Original German title (only if English translation exists and differs) -->
+{% if article.title_en and article.title_en != article.title %}
+<p style="margin: 0 0 12px 0; font-size: 13px; color: #999999; font-style: italic;">
+    Original: {{ article.title }}
+</p>
+{% endif %}
+```
+
+**Display Rules**:
+1. If `title_en` exists and differs from `title`: Show English as primary, German as subtitle
+2. If `title_en` is NULL or same as `title`: Show only the original title
+3. Subtitle styling: Smaller font (13px), gray color (#999999), italic
+
+## Error Handling
+
+### Translation Failure Scenarios
+
+| Scenario | Behavior | User Impact |
+|----------|----------|-------------|
+| Ollama server unavailable | Skip translation, continue with summarization | Newsletter shows German title only |
+| Translation timeout | Log error, store NULL in title_en | Newsletter shows German title only |
+| Empty title input | Return error immediately, skip API call | Newsletter shows German title only |
+| Ollama disabled in config | Skip translation entirely | Newsletter shows German title only |
+| Network error | Catch exception, log error, continue | Newsletter shows German title only |
+
+### Error Handling Principles
+
+1. **Non-blocking**: Translation failures never prevent article processing
+2. **Graceful degradation**: Fall back to original German title
+3. **Consistent logging**: All errors logged with descriptive messages
+4. **No retry logic**: Single attempt per article (same as summarization)
+5. **Silent failures**: Newsletter displays seamlessly regardless of translation status
+
+### Console Output Examples
+
+**Success Case**:
+```
+🔍 Crawling: Neuer U-Bahn-Ausbau in München geplant...
+   🌐 Translating title...
+   ✓ Title translated (0.8s)
+   🤖 Summarizing with AI...
+   ✓ Summary: 45 words (from 320 words, 2.3s)
+   ✓ Saved (320 words)
+```
+
+**Translation Failure Case**:
+```
+🔍 Crawling: Neuer U-Bahn-Ausbau in München geplant...
+   🌐 Translating title...
+   ⚠ Translation failed: Request timed out after 30 seconds
+   🤖 Summarizing with AI...
+   ✓ Summary: 45 words (from 320 words, 2.3s)
+   ✓ Saved (320 words)
+```
+
+## Testing Strategy
+
+### Unit Testing
+
+**Ollama Client Tests** (`test_ollama_client.py`):
+1. Test successful translation with valid German title
+2. Test empty title input handling
+3. Test timeout handling
+4. Test connection error handling
+5. Test response cleaning (quotes, newlines, whitespace)
+6. Test translation with special characters
+7. Test translation with very long titles
+
+**Test Data Examples**:
+- Simple: "München plant neue U-Bahn-Linie"
+- With quotes: "\"Historischer Tag\" für München"
+- With special chars: "Oktoberfest 2024: 7,5 Millionen Besucher"
+- Long: "Stadtrat beschließt umfassende Maßnahmen zur Verbesserung der Verkehrsinfrastruktur..."
+
+### Integration Testing
+
+**Crawler Service Tests**:
+1. Test article processing with translation enabled
+2. Test article processing with translation disabled
+3. Test article processing when translation fails
+4. Test database document structure includes new fields
+5. Test console output formatting
+
+### Manual Testing
+
+**End-to-End Workflow**:
+1. Enable Ollama in configuration
+2. Trigger crawl with `max_articles=2`
+3. Verify console shows translation status
+4. Check MongoDB for `title_en` and `translated_at` fields
+5. Send test newsletter
+6. Verify email displays English title with German subtitle
+
+**Test Scenarios**:
+- Fresh crawl with Ollama enabled
+- Re-crawl existing articles (should skip translation)
+- Crawl with Ollama disabled
+- Crawl with Ollama server stopped (simulate failure)
+
+### Performance Testing
+
+**Metrics to Monitor**:
+- Translation duration per article (target: < 2 seconds)
+- Impact on total crawl time (translation + summarization)
+- Ollama server resource usage
+
+**Expected Performance**:
+- Translation: ~0.5-1.5 seconds per title
+- Total per article: ~3-5 seconds (translation + summarization)
+- Acceptable for batch processing during scheduled crawls
+
+## Configuration
+
+### No New Configuration Required
+
+The translation feature uses existing Ollama configuration:
+
+```python
+# From config.py (existing)
+OLLAMA_ENABLED = True/False
+OLLAMA_BASE_URL = "http://ollama:11434"
+OLLAMA_MODEL = "phi3:latest"
+OLLAMA_TIMEOUT = 30
+```
+
+**Rationale**: Simplifies deployment and maintains consistency. Translation is automatically enabled/disabled with the existing `OLLAMA_ENABLED` flag.
+
+## Deployment Considerations
+
+### Docker Container Updates
+
+**Affected Services**:
+- `crawler` service: Needs rebuild to include new translation code
+- `sender` service: Needs rebuild to include updated newsletter template
+
+**Deployment Steps**:
+1. Update code in `news_crawler/ollama_client.py`
+2. Update code in `news_crawler/crawler_service.py`
+3. Update template in `news_sender/newsletter_template.html`
+4. Rebuild containers: `docker-compose up -d --build crawler sender`
+5. No database migration needed (new fields are nullable)
+
+### Backward Compatibility
+
+**Existing Articles**: Articles without `title_en` will display German title only (graceful fallback)
+
+**No Breaking Changes**: Newsletter template handles NULL `title_en` values
+
+### Rollback Plan
+
+If issues arise:
+1. Revert code changes
+2. Rebuild containers
+3. Existing articles with `title_en` will continue to work
+4. New articles will only have German titles
+
+## Future Enhancements
+
+### Potential Improvements (Out of Scope)
+
+1. **Batch Translation**: Translate multiple titles in single API call for efficiency
+2. **Translation Caching**: Cache common phrases/words to reduce API calls
+3. **Multi-language Support**: Add configuration for target language selection
+4. **Translation Quality Metrics**: Track and log translation quality scores
+5. **Retry Logic**: Implement retry with exponential backoff for failed translations
+6. **Admin API**: Add endpoint to re-translate existing articles
+
+These enhancements are not included in the current implementation to maintain simplicity and focus on core functionality.
--- a/.kiro/specs/article-title-translation/requirements.md
+++ b/.kiro/specs/article-title-translation/requirements.md
@@ -0,0 +1,75 @@
+# Requirements Document
+
+## Introduction
+
+This feature adds automatic translation of German article titles to English using the Ollama AI service. The translation will occur during the article crawling process and both the original German title and English translation will be stored in the database. The newsletter will display the English title prominently with the original German title as a subtitle when available.
+
+## Glossary
+
+- **Crawler Service**: The Python service that fetches articles from RSS feeds and processes them
+- **Ollama Client**: The Python client that communicates with the Ollama AI server for text processing
+- **Article Document**: The MongoDB document structure that stores article data
+- **Newsletter Template**: The HTML template used to render the email newsletter sent to subscribers
+- **Translation Result**: The response object returned by the Ollama translation function containing the translated title and metadata
+
+## Requirements
+
+### Requirement 1
+
+**User Story:** As a newsletter subscriber, I want to see article titles in English, so that I can quickly understand the content without knowing German
+
+#### Acceptance Criteria
+
+1. WHEN the Crawler Service processes an article, THE Ollama Client SHALL translate the German title to English
+2. THE Article Document SHALL store both the original German title and the English translation
+3. THE Newsletter Template SHALL display the English title as the primary heading
+4. WHERE an English translation exists, THE Newsletter Template SHALL display the original German title as a subtitle
+5. IF the translation fails, THEN THE Newsletter Template SHALL display the original German title as the primary heading
+
+### Requirement 2
+
+**User Story:** As a system administrator, I want translation to be integrated with the existing Ollama service, so that I don't need to configure additional services
+
+#### Acceptance Criteria
+
+1. THE Ollama Client SHALL provide a translate_title method that accepts a German title and returns an English translation
+2. THE translate_title method SHALL use the same Ollama server configuration as the existing summarization feature
+3. THE translate_title method SHALL use a temperature setting of 0.3 for consistent translations
+4. THE translate_title method SHALL limit the response to 100 tokens maximum for title-length outputs
+5. THE translate_title method SHALL return a Translation Result containing success status, translated title, error message, and duration
+
+### Requirement 3
+
+**User Story:** As a developer, I want translation errors to be handled gracefully, so that article processing continues even when translation fails
+
+#### Acceptance Criteria
+
+1. IF the Ollama server is unavailable, THEN THE Crawler Service SHALL continue processing articles without translations
+2. IF a translation request times out, THEN THE Crawler Service SHALL log the error and store the article with only the original title
+3. THE Crawler Service SHALL display translation status in the console output during crawling
+4. THE Article Document SHALL include a translated_at timestamp field when translation succeeds
+5. THE Article Document SHALL store NULL in the title_en field when translation fails
+
+### Requirement 4
+
+**User Story:** As a newsletter subscriber, I want translations to be accurate and natural, so that the English titles read fluently
+
+#### Acceptance Criteria
+
+1. THE Ollama Client SHALL provide a clear prompt instructing the model to translate German news headlines to English
+2. THE Ollama Client SHALL instruct the model to provide only the translation without explanations
+3. THE Ollama Client SHALL clean the translation output by removing quotes and extra text
+4. THE Ollama Client SHALL extract only the first line of the translation response
+5. THE Ollama Client SHALL trim whitespace from the translated title
+
+### Requirement 5
+
+**User Story:** As a system operator, I want to see translation performance metrics, so that I can monitor the translation feature effectiveness
+
+#### Acceptance Criteria
+
+1. THE Crawler Service SHALL log the translation duration for each article
+2. THE Crawler Service SHALL display a success indicator when translation completes
+3. THE Crawler Service SHALL display an error message when translation fails
+4. THE Translation Result SHALL include the duration in seconds
+5. THE Article Document SHALL store the translated_at timestamp for successful translations
--- a/.kiro/specs/article-title-translation/tasks.md
+++ b/.kiro/specs/article-title-translation/tasks.md
@@ -0,0 +1,47 @@
+# Implementation Plan
+
+- [x] 1. Add translate_title method to Ollama client
+  - Create the `translate_title()` method in `news_crawler/ollama_client.py` that accepts a title string and target language parameter
+  - Implement the translation prompt that instructs the model to translate German headlines to English without explanations
+  - Configure Ollama API call with temperature=0.3 and num_predict=100 for consistent title-length translations
+  - Implement response cleaning logic to remove quotes, extract first line only, and trim whitespace
+  - Add error handling for timeout, connection errors, HTTP errors, and empty title input
+  - Return a dictionary with success status, translated_title, error message, and duration fields
+  - _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 4.1, 4.2, 4.3, 4.4, 4.5_
+
+- [x] 2. Integrate translation into crawler service
+  - [x] 2.1 Add translation call in crawl_rss_feed function
+    - Locate the article processing section in `news_crawler/crawler_service.py` after content extraction
+    - Store the original title from article_data or entry
+    - Add conditional check for Config.OLLAMA_ENABLED before calling translation
+    - Call `ollama_client.translate_title()` with the original title
+    - Store the translation_result for later use in article document
+    - _Requirements: 1.1, 2.1_
+  
+  - [x] 2.2 Add console logging for translation status
+    - Add "🌐 Translating title..." message before translation call
+    - Add success message with duration: "✓ Title translated (X.Xs)"
+    - Add failure message with error: "⚠ Translation failed: {error}"
+    - _Requirements: 5.1, 5.2, 5.3_
+  
+  - [x] 2.3 Update article document structure
+    - Modify the article_doc dictionary to include `title_en` field with translated title or None
+    - Add `translated_at` field set to datetime.utcnow() on success or None on failure
+    - Ensure the original `title` field still contains the German title
+    - _Requirements: 1.2, 3.5_
+
+- [x] 3. Update newsletter template for bilingual title display
+  - Modify `news_sender/newsletter_template.html` to display English title as primary heading when available
+  - Add conditional logic to show original German title as subtitle only when English translation exists and differs
+  - Style the subtitle with smaller font (13px), gray color (#999999), and italic formatting
+  - Ensure fallback to German title when title_en is NULL or missing
+  - _Requirements: 1.3, 1.4, 1.5_
+
+- [x] 4. Test the translation feature end-to-end
+  - Rebuild the crawler Docker container with the new translation code
+  - Clear existing articles from the database for clean testing
+  - Trigger a test crawl with max_articles=2 to process fresh articles
+  - Verify console output shows translation status messages
+  - Check MongoDB to confirm title_en and translated_at fields are populated
+  - Send a test newsletter email to verify English titles display correctly with German subtitles
+  - _Requirements: 1.1, 1.2, 1.3, 1.4, 5.1, 5.2, 5.4, 5.5_