charieswei
/
TradingAgents
mirror of https://github.com/TauricResearch/TradingAgents.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
TradingAgents/docs/specs/news/tasks.md

35 KiB
Raw Blame History

News Domain Completion - Task Implementation Guide

Overview

Complete the final 5% of the news domain by implementing Dagster orchestration, OpenRouter-powered LLM sentiment analysis, vector embeddings, and RAG-powered semantic search. This builds on 95% complete infrastructure with PostgreSQL + TimescaleDB + pgvectorscale stack.

Total Estimated Time: 15-20 hours with AI assistance Target Completion: 4-5 days Test Coverage Requirement: Maintain >85% Architecture Pattern: Entity → Repository → Service → Dagster Op → Dagster Job

Implementation Phases

Phase 1: Entity Layer (2-3 hours)

Database and entity layer enhancements for LLM integration

Phase 2: Repository Layer (2-3 hours)

RAG-powered vector similarity search methods

Phase 3: LLM Integration (4-5 hours)

OpenRouter clients for sentiment and embeddings

Phase 4: Service Enhancement (2-3 hours)

Integrate LLM clients into NewsService workflow

Phase 5: Dagster Orchestration (3-4 hours)

Jobs, ops, schedules, and sensors for automated collection

Phase 6: Testing & Documentation (2-3 hours)

Integration tests, performance validation, and documentation updates

Task Breakdown

Phase 1: Entity Layer

T001: Enhance NewsArticle Dataclass - Sentiment Fields

Priority: Critical | Duration: 1-2 hours | Dependencies: None

Description: Add LLM sentiment fields to existing NewsArticle dataclass

Acceptance Criteria:

  • Add sentiment_confidence: Optional[float] field (0.0-1.0 range)
  • Add sentiment_label: Optional[str] field ("positive", "negative", "neutral")
  • Update to_entity() method to include new sentiment fields
  • Update from_entity() method to populate new sentiment fields
  • Add has_reliable_sentiment() helper method (confidence >= 0.6)

Implementation Details:

@dataclass
class NewsArticle:
    # Existing fields...
    sentiment_score: Optional[float] = None  # Already exists

    # New LLM sentiment fields
    sentiment_confidence: Optional[float] = None  # 0.0 to 1.0
    sentiment_label: Optional[str] = None  # "positive", "negative", "neutral"

    # Vector fields already exist from 95% complete infrastructure
    title_embedding: Optional[List[float]] = None
    content_embedding: Optional[List[float]] = None

    def has_reliable_sentiment(self) -> bool:
        """Check if sentiment analysis is reliable."""
        return bool(
            self.sentiment_score is not None
            and self.sentiment_confidence is not None
            and self.sentiment_confidence >= 0.6
        )

Files to Modify:

  • /Users/martinrichards/code/TradingAgents/tradingagents/domains/news/news_repository.py (NewsArticle dataclass section)

Test Requirements:

  • Dataclass instantiation with new fields
  • to_entity() and from_entity() roundtrip conversion
  • has_reliable_sentiment() validation logic
  • Edge cases (None values, boundary conditions)

T002: Database Migration - Sentiment Fields

Priority: Critical | Duration: 1 hour | Dependencies: T001

Description: Create Alembic migration to add sentiment fields to news_articles table

Acceptance Criteria:

  • Create Alembic migration script add_sentiment_fields.py
  • Add sentiment_confidence FLOAT column (nullable)
  • Add sentiment_label VARCHAR(20) column (nullable)
  • Add index on sentiment_label for filtering
  • Migration tested with upgrade and downgrade
  • Rollback capability verified

Implementation Details:

# alembic/versions/20250111_add_sentiment_fields.py
def upgrade():
    op.add_column('news_articles', sa.Column('sentiment_confidence', sa.Float(), nullable=True))
    op.add_column('news_articles', sa.Column('sentiment_label', sa.String(20), nullable=True))
    op.create_index('idx_news_sentiment_label', 'news_articles', ['sentiment_label'])

def downgrade():
    op.drop_index('idx_news_sentiment_label', table_name='news_articles')
    op.drop_column('news_articles', 'sentiment_label')
    op.drop_column('news_articles', 'sentiment_confidence')

Files to Create:

  • /Users/martinrichards/code/TradingAgents/alembic/versions/20250111_add_sentiment_fields.py

Test Requirements:

  • Migration upgrade succeeds
  • Migration downgrade succeeds
  • Index is created properly
  • Existing data remains intact

Phase 2: Repository Layer

T003: NewsRepository - Vector Similarity Search

Priority: Critical | Duration: 2-3 hours | Dependencies: T001, T002

Description: Add RAG-powered vector similarity search using pgvectorscale

Acceptance Criteria:

  • Implement find_similar_articles() method with cosine distance
  • Support similarity threshold filtering (0.0-1.0)
  • Support optional symbol filtering
  • Results ordered by similarity descending
  • Proper async/await with session management
  • Logging for debugging and monitoring

Implementation Details:

async def find_similar_articles(
    self,
    embedding: List[float],
    limit: int = 10,
    threshold: float = 0.7,
    symbol: Optional[str] = None
) -> List[NewsArticle]:
    """
    Find articles similar to given embedding using pgvectorscale cosine distance.

    pgvectorscale operator: <=> for cosine distance
    Cosine similarity = 1 - cosine_distance
    """
    async with self.db_manager.get_session() as session:
        # Build query with vector similarity
        query = select(
            NewsArticleEntity,
            (1 - NewsArticleEntity.title_embedding.cosine_distance(embedding)).label('similarity')
        ).filter(
            NewsArticleEntity.title_embedding.is_not(None)
        )

        # Optional symbol filter
        if symbol:
            query = query.filter(NewsArticleEntity.symbol == symbol)

        # Filter by similarity threshold and order by distance
        query = query.filter(
            (1 - NewsArticleEntity.title_embedding.cosine_distance(embedding)) >= threshold
        ).order_by(
            NewsArticleEntity.title_embedding.cosine_distance(embedding)
        ).limit(limit)

        result = await session.execute(query)
        rows = result.all()

        articles = [NewsArticle.from_entity(row[0]) for row in rows]
        logger.info(f"Found {len(articles)} similar articles (threshold={threshold})")
        return articles

Files to Modify:

  • /Users/martinrichards/code/TradingAgents/tradingagents/domains/news/news_repository.py (add method to NewsRepository class)

Test Requirements:

  • Vector similarity returns correct results with test data
  • Similarity threshold filtering works correctly
  • Symbol filtering works correctly
  • Empty result handling
  • Performance test (<1s for typical queries)

T004: NewsRepository - Batch Embedding Updates

Priority: Medium | Duration: 1 hour | Dependencies: T003

Description: Add efficient batch embedding update method

Acceptance Criteria:

  • Implement batch_update_embeddings() method
  • Use PostgreSQL bulk update operations
  • Support title and content embeddings
  • Update timestamp on modification
  • Return count of updated articles

Implementation Details:

async def batch_update_embeddings(
    self,
    article_embeddings: List[Tuple[UUID, List[float], List[float]]]
) -> int:
    """Efficiently batch update embeddings for multiple articles."""
    if not article_embeddings:
        return 0

    async with self.db_manager.get_session() as session:
        stmt = update(NewsArticleEntity).where(
            NewsArticleEntity.id == bindparam('article_id')
        ).values(
            title_embedding=bindparam('title_emb'),
            content_embedding=bindparam('content_emb'),
            updated_at=func.now()
        )

        batch_data = [
            {
                'article_id': article_id,
                'title_emb': title_emb,
                'content_emb': content_emb
            }
            for article_id, title_emb, content_emb in article_embeddings
        ]

        await session.execute(stmt, batch_data)
        logger.info(f"Batch updated embeddings for {len(article_embeddings)} articles")
        return len(article_embeddings)

Files to Modify:

  • /Users/martinrichards/code/TradingAgents/tradingagents/domains/news/news_repository.py

Test Requirements:

  • Batch update modifies correct articles
  • Performance test (sub-second for 50 articles)
  • Empty list handling
  • Database rollback on errors

Phase 3: LLM Integration

T005: OpenRouter Sentiment Client

Priority: Critical | Duration: 2-3 hours | Dependencies: T001

Description: Implement OpenRouter client for LLM sentiment analysis

Acceptance Criteria:

  • OpenRouter API integration using quick_think_llm (claude-3.5-haiku)
  • Structured JSON output: score, confidence, label, reasoning
  • Financial news-focused prompts
  • Exponential backoff retry logic (3 attempts)
  • Keyword-based fallback on API failures
  • Proper error handling and logging

Implementation Details:

@dataclass
class SentimentResult:
    """Result from sentiment analysis."""
    score: float  # -1.0 to 1.0
    confidence: float  # 0.0 to 1.0
    label: str  # "positive", "negative", "neutral"
    reasoning: str

class OpenRouterSentimentClient:
    """Client for sentiment analysis via OpenRouter."""

    def __init__(self, config: TradingAgentsConfig):
        self.api_key = config.openrouter_api_key
        self.model = config.quick_think_llm  # claude-3.5-haiku
        self.base_url = "https://openrouter.ai/api/v1/chat/completions"

    async def analyze_sentiment(self, title: str, content: str) -> SentimentResult:
        """Analyze sentiment with fallback to keyword-based analysis."""
        try:
            prompt = self._build_sentiment_prompt(title, content)
            response = await self._call_openrouter(prompt)
            return self._parse_sentiment_response(response)
        except Exception as e:
            logger.warning(f"OpenRouter sentiment failed: {e}, using fallback")
            return self._fallback_sentiment(title, content)

    def _fallback_sentiment(self, title: str, content: str) -> SentimentResult:
        """Keyword-based fallback for sentiment analysis."""
        text = f"{title} {content}".lower()
        positive_keywords = ['gain', 'up', 'rise', 'growth', 'profit', 'beat']
        negative_keywords = ['loss', 'down', 'fall', 'decline', 'miss', 'concern']

        pos_count = sum(1 for kw in positive_keywords if kw in text)
        neg_count = sum(1 for kw in negative_keywords if kw in text)

        if pos_count > neg_count:
            return SentimentResult(0.3, 0.5, "positive", "Keyword-based fallback")
        elif neg_count > pos_count:
            return SentimentResult(-0.3, 0.5, "negative", "Keyword-based fallback")
        else:
            return SentimentResult(0.0, 0.5, "neutral", "Keyword-based fallback")

Files to Create:

  • /Users/martinrichards/code/TradingAgents/tradingagents/domains/news/clients/openrouter_sentiment_client.py

Test Requirements:

  • API response parsing tests with VCR
  • Retry logic tests
  • Fallback mechanism tests
  • Error handling tests
  • Integration test with real API (optional)

T006: OpenRouter Embeddings Client

Priority: Critical | Duration: 1-2 hours | Dependencies: T001

Description: Implement OpenRouter client for vector embeddings generation

Acceptance Criteria:

  • OpenRouter embeddings API integration (text-embedding-ada-002)
  • Text preprocessing (8000 char limit)
  • Batch processing support for multiple texts
  • 1536-dimensional vector validation
  • Zero-vector fallback on API failures
  • Proper error handling and logging

Implementation Details:

class OpenRouterEmbeddingsClient:
    """Client for generating embeddings via OpenRouter."""

    def __init__(self, config: TradingAgentsConfig):
        self.api_key = config.openrouter_api_key
        self.model = "openai/text-embedding-ada-002"  # Via OpenRouter
        self.base_url = "https://openrouter.ai/api/v1/embeddings"

    async def generate_embeddings(self, texts: List[str]) -> List[List[float]]:
        """Generate 1536-dim embeddings for multiple texts."""
        if not texts:
            return []

        try:
            processed_texts = [self._preprocess_text(text) for text in texts]

            headers = {
                "Authorization": f"Bearer {self.api_key}",
                "Content-Type": "application/json"
            }

            payload = {"model": self.model, "input": processed_texts}

            async with aiohttp.ClientSession() as session:
                async with session.post(
                    self.base_url,
                    headers=headers,
                    json=payload,
                    timeout=aiohttp.ClientTimeout(total=60)
                ) as response:
                    response.raise_for_status()
                    data = await response.json()
                    embeddings = [item['embedding'] for item in data['data']]

                    # Validate dimensions
                    for i, emb in enumerate(embeddings):
                        if len(emb) != 1536:
                            raise ValueError(f"Invalid embedding dimension: {len(emb)}")

                    return embeddings

        except Exception as e:
            logger.error(f"Embeddings generation failed: {e}, using zero vectors")
            return [[0.0] * 1536 for _ in texts]

    async def generate_article_embeddings(
        self,
        article: NewsArticle
    ) -> Tuple[List[float], List[float]]:
        """Generate embeddings for article title and content."""
        texts = []
        if article.headline:
            texts.append(article.headline)
        if article.summary:
            combined = f"{article.headline} {article.summary}"
            texts.append(combined)

        if not texts:
            return [0.0] * 1536, [0.0] * 1536

        embeddings = await self.generate_embeddings(texts)
        title_embedding = embeddings[0] if len(embeddings) > 0 else [0.0] * 1536
        content_embedding = embeddings[1] if len(embeddings) > 1 else [0.0] * 1536

        return title_embedding, content_embedding

    def _preprocess_text(self, text: str) -> str:
        """Preprocess text for optimal embedding generation."""
        cleaned = " ".join(text.split())
        return cleaned[:8000]  # OpenAI embedding limit

Files to Create:

  • /Users/martinrichards/code/TradingAgents/tradingagents/domains/news/clients/openrouter_embeddings_client.py

Test Requirements:

  • API response parsing tests with VCR
  • Batch processing tests
  • Vector dimension validation tests
  • Text preprocessing tests
  • Zero-vector fallback tests

T007: Enhance NewsService - LLM Integration

Priority: Critical | Duration: 2-3 hours | Dependencies: T005, T006

Description: Integrate OpenRouter LLM clients into NewsService workflow

Acceptance Criteria:

  • Add LLM clients to NewsService __init__()
  • Implement _enrich_articles() method for LLM processing
  • Update update_company_news() to call enrichment
  • Implement find_similar_news() for RAG queries
  • Best-effort processing (failures don't block storage)
  • Proper error handling and logging

Implementation Details:

class NewsService:
    def __init__(
        self,
        google_client: GoogleNewsClient,
        repository: NewsRepository,
        article_scraper: ArticleScraperClient,
        sentiment_client: OpenRouterSentimentClient,
        embeddings_client: OpenRouterEmbeddingsClient,
    ):
        self.google_client = google_client
        self.repository = repository
        self.article_scraper = article_scraper
        self.sentiment_client = sentiment_client
        self.embeddings_client = embeddings_client

    async def update_company_news(self, symbol: str) -> NewsUpdateResult:
        """
        Update company news with full LLM enrichment pipeline.

        Flow: RSS → Scrape → LLM Sentiment → Embeddings → Store
        """
        # 1. Get RSS feed
        google_articles = self.google_client.get_company_news(symbol)

        # 2. Scrape content
        scraped_articles = await self._scrape_articles(google_articles)

        # 3. Enrich with LLM (sentiment + embeddings)
        enriched_articles = await self._enrich_articles(scraped_articles)

        # 4. Store in repository
        stored_articles = await self.repository.upsert_batch(enriched_articles, symbol)

        return NewsUpdateResult(...)

    async def _enrich_articles(
        self,
        articles: List[NewsArticle]
    ) -> List[NewsArticle]:
        """Enrich articles with LLM sentiment and vector embeddings."""
        enriched = []

        for article in articles:
            try:
                # Generate sentiment
                sentiment_result = await self.sentiment_client.analyze_sentiment(
                    article.headline,
                    article.summary or ""
                )

                article.sentiment_score = sentiment_result.score
                article.sentiment_confidence = sentiment_result.confidence
                article.sentiment_label = sentiment_result.label

                # Generate embeddings
                title_emb, content_emb = await self.embeddings_client.generate_article_embeddings(article)
                article.title_embedding = title_emb
                article.content_embedding = content_emb

                enriched.append(article)

            except Exception as e:
                logger.warning(f"Failed to enrich article {article.url}: {e}")
                enriched.append(article)  # Store without enrichment

        return enriched

    async def find_similar_news(
        self,
        query_text: str,
        symbol: Optional[str] = None,
        limit: int = 5
    ) -> List[NewsArticle]:
        """Find news articles similar to query text using RAG vector search."""
        # Generate embedding for query
        query_embeddings = await self.embeddings_client.generate_embeddings([query_text])
        query_embedding = query_embeddings[0]

        # Search for similar articles
        similar_articles = await self.repository.find_similar_articles(
            embedding=query_embedding,
            limit=limit,
            threshold=0.7,
            symbol=symbol
        )

        return similar_articles

Files to Modify:

  • /Users/martinrichards/code/TradingAgents/tradingagents/domains/news/news_service.py

Test Requirements:

  • Mock LLM clients for unit tests
  • Integration test with real services
  • Error handling and fallback tests
  • Performance test for batch enrichment

Phase 4: Dagster Orchestration

T008: Dagster Directory Structure

Priority: High | Duration: 30 minutes | Dependencies: None

Description: Create directory structure for Dagster jobs, ops, and schedules

Acceptance Criteria:

  • Create tradingagents/data/ directory
  • Create subdirectories: jobs/, ops/, schedules/, sensors/
  • Create __init__.py files for all directories
  • Import structure allows clean imports

Implementation Details:

tradingagents/data/
├── __init__.py
├── jobs/
│   ├── __init__.py
│   └── news_collection.py
├── ops/
│   ├── __init__.py
│   └── news_ops.py
├── schedules/
│   ├── __init__.py
│   └── news_schedules.py
└── sensors/
    ├── __init__.py
    └── news_sensors.py

Files to Create:

  • All directory and __init__.py files above

Test Requirements:

  • Import tests for all modules
  • Directory structure validation

T009: Dagster Ops - News Collection

Priority: High | Duration: 2-3 hours | Dependencies: T007, T008

Description: Implement Dagster op for news collection per symbol

Acceptance Criteria:

  • collect_news_for_symbol op implemented
  • Proper resource management (database_manager)
  • Error handling and logging
  • Output metadata (articles_found, articles_scraped, etc.)
  • Retry policy configured
  • Op tested with build_op_context

Implementation Details:

# tradingagents/data/ops/news_ops.py
from dagster import op, OpExecutionContext, Out, RetryPolicy

@op(
    required_resource_keys={"database_manager"},
    out=Out(dict),
    tags={"kind": "news", "domain": "news"},
    retry_policy=RetryPolicy(max_retries=3, delay=10, backoff=BackoffPolicy.EXPONENTIAL),
)
def collect_news_for_symbol(context: OpExecutionContext, symbol: str) -> dict:
    """
    Collect and process news for a single stock symbol.

    Returns dict with collection statistics.
    """
    context.log.info(f"Starting news collection for {symbol}")

    try:
        config = TradingAgentsConfig.from_env()
        db_manager = context.resources.database_manager
        news_service = NewsService.build(db_manager, config)

        result = await news_service.update_company_news(symbol)

        context.log.info(f"Completed: {result.articles_scraped} articles for {symbol}")

        return {
            "symbol": symbol,
            "articles_found": result.articles_found,
            "articles_scraped": result.articles_scraped,
            "articles_failed": result.articles_failed,
            "status": result.status,
        }

    except Exception as e:
        context.log.error(f"News collection failed for {symbol}: {e}")
        raise

Files to Create:

  • /Users/martinrichards/code/TradingAgents/tradingagents/data/ops/news_ops.py

Test Requirements:

  • Op execution tests with mock resources
  • Error handling tests
  • Retry logic tests
  • Metadata validation tests

T010: Dagster Job - Daily News Collection

Priority: High | Duration: 1-2 hours | Dependencies: T009

Description: Implement Dagster job that orchestrates news collection across symbols

Acceptance Criteria:

  • news_collection_daily job implemented
  • Dynamic op mapping for parallel symbol processing
  • Proper job tags and metadata
  • Configuration for symbol list
  • Job tested with execute_in_process

Implementation Details:

# tradingagents/data/jobs/news_collection.py
from dagster import job, DynamicOut, DynamicOutput, OpExecutionContext, op
from tradingagents.data.ops.news_ops import collect_news_for_symbol

@op(out=DynamicOut())
def get_symbols_to_collect(context: OpExecutionContext) -> Generator[DynamicOutput, None, None]:
    """Get list of symbols to collect news for from config."""
    symbols = context.op_config.get("symbols", ["AAPL", "GOOGL", "MSFT", "TSLA"])
    context.log.info(f"Collecting news for {len(symbols)} symbols: {symbols}")

    for symbol in symbols:
        yield DynamicOutput(symbol, mapping_key=symbol)

@job(tags={"dagster/priority": "high", "domain": "news"})
def news_collection_daily():
    """
    Daily news collection job for all configured symbols.

    Workflow:
    1. Get symbols to collect
    2. Fan out: collect news for each symbol in parallel
    3. Aggregate results
    """
    get_symbols_to_collect().map(collect_news_for_symbol)

Files to Create:

  • /Users/martinrichards/code/TradingAgents/tradingagents/data/jobs/news_collection.py

Test Requirements:

  • Job execution tests
  • Dynamic mapping tests
  • Configuration tests
  • Parallel execution validation

T011: Dagster Schedule - Daily Trigger

Priority: High | Duration: 1 hour | Dependencies: T010

Description: Implement Dagster schedule for daily news collection at 6 AM UTC

Acceptance Criteria:

  • news_collection_daily_schedule schedule implemented
  • Cron expression: 0 6 * * * (daily at 6 AM UTC)
  • RunRequest configuration with symbol list
  • Proper tags and metadata
  • Schedule tested with evaluate_tick

Implementation Details:

# tradingagents/data/schedules/news_schedules.py
from dagster import schedule, ScheduleEvaluationContext, RunRequest
from tradingagents.data.jobs.news_collection import news_collection_daily

@schedule(
    job=news_collection_daily,
    cron_schedule="0 6 * * *",  # Daily at 6 AM UTC
    execution_timezone="UTC",
)
def news_collection_daily_schedule(context: ScheduleEvaluationContext):
    """Schedule for daily news collection at 6 AM UTC."""
    return RunRequest(
        run_key=f"news_collection_{context.scheduled_execution_time.isoformat()}",
        run_config={
            "ops": {
                "get_symbols_to_collect": {
                    "config": {
                        "symbols": ["AAPL", "GOOGL", "MSFT", "TSLA", "AMZN", "META", "NVDA"]
                    }
                }
            }
        },
        tags={
            "scheduled_time": context.scheduled_execution_time.isoformat(),
            "job_type": "news_collection",
        },
    )

Files to Create:

  • /Users/martinrichards/code/TradingAgents/tradingagents/data/schedules/news_schedules.py

Test Requirements:

  • Schedule evaluation tests
  • Cron schedule validation
  • RunRequest configuration tests
  • Timezone handling tests

T012: Dagster Sensor - Failure Alerting

Priority: Medium | Duration: 1 hour | Dependencies: T010

Description: Implement Dagster sensor for job failure alerting

Acceptance Criteria:

  • news_collection_failure_sensor run failure sensor implemented
  • Monitors news_collection_daily job
  • Logs failure details
  • Placeholder for external alerting (Slack, PagerDuty, etc.)
  • Sensor tested with run failure events

Implementation Details:

# tradingagents/data/sensors/news_sensors.py
from dagster import run_failure_sensor, RunFailureSensorContext
from tradingagents.data.jobs.news_collection import news_collection_daily

@run_failure_sensor(
    name="news_collection_failure_sensor",
    monitored_jobs=[news_collection_daily],
)
def news_collection_failure_alert(context: RunFailureSensorContext):
    """Alert when news collection job fails."""
    context.log.error(
        f"News collection job failed!\n"
        f"Run ID: {context.dagster_run.run_id}\n"
        f"Failure: {context.failure_event.event_specific_data}"
    )

    # TODO: Implement external alerting
    # send_slack_alert(...)
    # send_pagerduty_alert(...)

Files to Create:

  • /Users/martinrichards/code/TradingAgents/tradingagents/data/sensors/news_sensors.py

Test Requirements:

  • Sensor evaluation tests
  • Failure detection tests
  • Logging validation tests

Phase 5: Testing & Documentation

T013: Integration Tests - End-to-End Workflow

Priority: High | Duration: 2-3 hours | Dependencies: T007, T010

Description: Comprehensive integration tests for complete news domain workflow

Acceptance Criteria:

  • End-to-end workflow test: RSS → Scrape → LLM → Vector → Store
  • RAG query test: Vector similarity search with semantic matching
  • AgentToolkit integration test
  • Performance tests (< 2s queries, < 1s vector search)
  • Error recovery and fallback tests
  • Test coverage maintained above 85%

Implementation Details:

# tests/domains/news/integration/test_news_workflow.py

@pytest.mark.asyncio
async def test_complete_news_pipeline_end_to_end(test_db_manager):
    """Test complete pipeline: RSS → Scrape → LLM → Vector → Store."""
    config = TradingAgentsConfig.from_test_env()
    service = NewsService.build(test_db_manager, config)

    # Execute full pipeline
    result = await service.update_company_news("AAPL")

    # Verify results
    assert result.status == "completed"
    assert result.articles_scraped > 0

    # Verify database storage
    articles = await service.repository.list_by_date_range(
        symbol="AAPL",
        start_date=date.today(),
        end_date=date.today()
    )

    assert len(articles) > 0

    # Verify LLM enrichment
    for article in articles:
        assert article.sentiment_score is not None
        assert article.sentiment_confidence is not None
        assert article.title_embedding is not None
        assert len(article.title_embedding) == 1536

@pytest.mark.asyncio
async def test_rag_vector_similarity_search(test_db_manager):
    """Test RAG vector similarity search functionality."""
    service = NewsService.build(test_db_manager, TradingAgentsConfig.from_test_env())

    # Find similar articles
    similar_articles = await service.find_similar_news(
        query_text="Apple earnings beat expectations",
        symbol="AAPL",
        limit=5
    )

    assert len(similar_articles) <= 5
    # Verify articles are relevant (high similarity scores)

@pytest.mark.asyncio
async def test_performance_benchmarks(test_db_manager):
    """Test performance meets requirements."""
    repository = NewsRepository(test_db_manager)

    # Test query performance (< 2s requirement)
    start_time = time.time()
    articles = await repository.list_by_date_range(
        symbol="AAPL",
        start_date=date.today() - timedelta(days=30),
        end_date=date.today()
    )
    query_time = time.time() - start_time

    assert query_time < 2.0, f"Query took {query_time}s, should be < 2s"

    # Test vector similarity performance (< 1s requirement)
    test_embedding = [0.1] * 1536
    start_time = time.time()
    similar = await repository.find_similar_articles(test_embedding, limit=10)
    vector_time = time.time() - start_time

    assert vector_time < 1.0, f"Vector search took {vector_time}s, should be < 1s"

Files to Create:

  • /Users/martinrichards/code/TradingAgents/tests/domains/news/integration/test_news_workflow.py

Test Requirements:

  • All integration tests pass
  • Performance benchmarks met
  • Test coverage > 85%

T014: Dagster Tests

Priority: Medium | Duration: 1 hour | Dependencies: T010, T011

Description: Unit tests for Dagster ops, jobs, and schedules

Acceptance Criteria:

  • Op execution tests with mocked resources
  • Job execution tests
  • Schedule evaluation tests
  • Error handling tests
  • All Dagster components tested

Implementation Details:

# tests/data/ops/test_news_ops.py
from dagster import build_op_context
from tradingagents.data.ops.news_ops import collect_news_for_symbol

def test_collect_news_for_symbol_op():
    """Test Dagster op for news collection."""
    context = build_op_context(
        resources={"database_manager": mock_database_manager}
    )

    result = collect_news_for_symbol(context, "AAPL")

    assert result["symbol"] == "AAPL"
    assert result["status"] == "completed"
    assert result["articles_found"] >= 0

# tests/data/jobs/test_news_collection.py
from dagster import execute_in_process
from tradingagents.data.jobs.news_collection import news_collection_daily

def test_news_collection_daily_job():
    """Test Dagster job execution."""
    result = execute_in_process(
        news_collection_daily,
        run_config={
            "ops": {
                "get_symbols_to_collect": {
                    "config": {"symbols": ["AAPL"]}
                }
            }
        }
    )

    assert result.success

Files to Create:

  • /Users/martinrichards/code/TradingAgents/tests/data/ops/test_news_ops.py
  • /Users/martinrichards/code/TradingAgents/tests/data/jobs/test_news_collection.py
  • /Users/martinrichards/code/TradingAgents/tests/data/schedules/test_news_schedules.py

Test Requirements:

  • All Dagster tests pass
  • Coverage > 85% for Dagster code

T015: Documentation Updates

Priority: Medium | Duration: 1-2 hours | Dependencies: T013, T014

Description: Update documentation and monitoring for new functionality

Acceptance Criteria:

  • Update API documentation for new methods
  • Dagster job configuration examples
  • Performance monitoring queries
  • Troubleshooting guide for common issues
  • AgentToolkit integration documentation
  • README updates

Files to Modify:

  • /Users/martinrichards/code/TradingAgents/docs/domains/news.md
  • /Users/martinrichards/code/TradingAgents/docs/api-reference.md
  • /Users/martinrichards/code/TradingAgents/README.md

Test Requirements:

  • Documentation accuracy validation
  • Configuration example testing
  • Link validation

Parallel Development Opportunities

AI Agent Collaboration Points

Tasks T005 & T006 can be developed in parallel:

  • Both are independent OpenRouter client implementations
  • Different LLM capabilities (sentiment vs embeddings)
  • Can be tested independently with pytest-vcr

Tasks T009, T010, T011 can be developed in parallel after T008:

  • Ops, jobs, and schedules are independent components
  • Can be tested separately
  • Integration testing happens in T014

Critical Path Analysis

Critical Path: T001 → T002 → T003 → T007 → T009 → T010 → T013

Parallel Branches:

  1. LLM Clients: T005 + T006 (parallel with T003-T004)
  2. Dagster Components: T009 + T010 + T011 (after T008)
  3. Testing: Unit tests alongside implementation

Success Metrics

Technical Metrics:

  • Test coverage >85% maintained
  • Query performance <2s for 30-day lookback
  • Vector search performance <1s for top-10 results
  • Zero breaking changes to AgentToolkit
  • Dagster jobs execute successfully

Functional Metrics:

  • OpenRouter LLM sentiment analysis operational
  • Vector embeddings enable semantic search
  • Dagster schedules running daily without failures
  • Agent context enriched with sentiment and similarity

Quality Metrics:

  • All acceptance criteria met for each task
  • Comprehensive error handling and fallbacks
  • Production-ready monitoring via Dagster UI
  • Complete documentation for all new features

Implementation Guidelines

TDD Approach

Every task follows: Write test → Write code → Refactor

Layered Architecture Pattern

Strict adherence to: Entity → Repository → Service → Dagster Op → Dagster Job

Error Handling Strategy

Graceful fallbacks for all LLM API dependencies (keyword sentiment, zero vectors)

Performance Requirements

Async operations with proper connection pooling throughout

Testing Strategy

Unit tests + Integration tests + pytest-vcr for external API calls

Risk Mitigation Strategies

LLM API Dependencies

  • Implement comprehensive fallback strategies
  • Use pytest-vcr for deterministic testing
  • Mock clients for unit tests
  • Monitor API costs and rate limits

Database Performance

  • Test with realistic data volumes
  • Monitor query performance during development
  • Use proper indexes for vector operations
  • Regular performance profiling

Dagster Integration

  • Start with simple ops and jobs
  • Test incrementally before full integration
  • Use Dagster UI for debugging
  • Implement comprehensive logging

This comprehensive task breakdown provides clear implementation guidance for completing the final 5% of the news domain while maintaining architectural consistency with Dagster orchestration and leveraging AI-assisted development patterns.