TradingAgents/docs/agent/decisions/009-industry-deep-dive-quality.md

type	status	date	agent_author	tags	related_files	pr
decision	active	2026-03-17	copilot+claude	scanner industry-deep-dive tool-execution prompt-engineering yfinance	tradingagents/agents/scanners/industry_deep_dive.py tradingagents/agents/utils/tool_runner.py tradingagents/agents/utils/scanner_tools.py tradingagents/dataflows/yfinance_scanner.py	13

Context

Phase 2 (Industry Deep Dive) produced sparse reports despite receiving ~21K chars of Phase 1 context. Three root causes were identified:

1. LLM guessing sector keys — the LLM had to infer valid sector_key strings (e.g., "financial-services" vs "financials") with no guidance, leading to failed tool calls.
2. Thin industry data — get_industry_performance_yfinance returned only static metadata (name, rating, market weight). No performance signal for the LLM to act on.
3. Tool-call skipping under long context — weaker local LLMs (Ollama/qwen) sometimes produce a short prose response instead of calling tools when the prompt is long.

The Decision

Three-pronged fix (PR #13):

1. Enriched Industry Performance Data

get_industry_performance_yfinance now batch-downloads 1-month price history for the top 10 tickers in each industry and computes 1-day, 1-week, and 1-month percentage returns. Output table expands from 4 to 7 columns:

| Company | Symbol | Rating | Market Weight | 1-Day % | 1-Week % | 1-Month % |

Both download and display use head(10) for consistency (avoids N/A rows for positions 11-20).

2. Explicit Sector Routing via _extract_top_sectors()

industry_deep_dive.py defines:

• VALID_SECTOR_KEYS — the 11 canonical yfinance sector key strings
• _DISPLAY_TO_KEY — maps display names (e.g., "Financial Services") to keys (e.g., "financial-services")
• _extract_top_sectors(sector_report, n) — parses the Phase 1 sector performance table, ranks sectors by absolute 1-month move, returns top-N valid keys

The prompt now injects the pre-extracted keys directly:

Call get_industry_performance for EACH of these top sectors: 'energy', 'communication-services', 'technology'
Valid sector_key values: 'technology', 'healthcare', 'financial-services', ...

This eliminates LLM guesswork entirely.

3. Tool-Call Nudge in run_tool_loop

If the LLM's first response has no tool_calls and is under 500 characters, a HumanMessage nudge is appended before re-invoking. Fires once only to avoid loops. Prevents short-circuit prose responses from weak LLMs under heavy context.

4. Tool Description Update

get_industry_performance docstring now enumerates all 11 valid sector keys so they appear in the tool schema visible to the LLM.

Constraints

• _extract_top_sectors() must degrade gracefully: if parsing fails (malformed Phase 1 report), it falls back to the top 3 default sectors ["technology", "financial-services", "energy"].
• The tool-call nudge fires at most once per agent invocation — do not loop on nudge.
• get_industry_performance_yfinance must use head(10) for both download and display to prevent N/A rows (Mistake #11: was displaying 20 rows but only downloading data for 10).

Actionable Rules

• Always inject pre-extracted sector keys into Industry Deep Dive prompt — never rely on the LLM to guess valid sector_key values.
• When enriching get_industry_performance_yfinance, keep download count and display count in sync.
• Tool-call nudge threshold is 500 chars — do not raise it; the intent is to catch short non-tool responses, not legitimate brief answers.
• All 11 VALID_SECTOR_KEYS must be listed in the get_industry_performance tool docstring.

Tests Added

15 new tests in tests/test_industry_deep_dive.py:

• 8 tests for _extract_top_sectors() parsing and edge cases
• 4 tests for nudge mechanism (mock chain)
• 3 tests for enriched output format (network-dependent, auto-skip if offline)