diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 00000000..86c7da49 --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1,16 @@ +{ + "hooks": { + "PostToolUse": [ + { + "matcher": "Bash", + "hooks": [ + { + "type": "prompt", + "prompt": "Check if the Bash command that just ran was a git commit (look for 'git commit' in the tool input). If it was NOT a git commit, do nothing — return empty output. If it WAS a git commit, remind the agent: 'A commit was just made. Update docs/agent/CURRENT_STATE.md to reflect what was committed — add to Recent Progress and remove any resolved blockers. Do NOT run a full memory rebuild, just update the state file.'", + "timeout": 10 + } + ] + } + ] + } +} diff --git a/.claude/skills/architecture-coordinator/SKILL.md b/.claude/skills/architecture-coordinator/SKILL.md deleted file mode 100644 index 6c2b1374..00000000 --- a/.claude/skills/architecture-coordinator/SKILL.md +++ /dev/null @@ -1,147 +0,0 @@ ---- -name: Architecture-First Reading Protocol -description: > - This skill should be used at the start of every new technical task, new session, - or when switching to a different part of the codebase. It enforces mandatory reading - of architectural decisions, current project state, and active plans before any code - is written, modified, or proposed. Relevant when the user says "implement a feature", - "fix a bug", "refactor code", "add a new module", "modify configuration", "change architecture", - "start a task", "begin work on", "let's build", or "work on". This skill acts as a gatekeeper - ensuring all code changes respect established Architecture Decision Records (ADRs). -version: 0.1.0 ---- - -# Architecture-First Reading Protocol - -## Purpose - -Enforce a mandatory reading sequence before writing any code, modifying configurations, -or proposing solutions. All established architectural rules in `docs/agent/decisions/` -are treated as absolute laws. Violating an ADR without explicit user approval is forbidden. - -## Mandatory Reading Sequence - -Execute the following steps **in order** before producing any code or solution. - -### Step 1: Read Current State - -Read `docs/agent/CURRENT_STATE.md` to understand: - -- The active milestone and sprint focus -- Any blockers or constraints currently in effect -- Recent changes that affect the working context - -If the file does not exist, note this and proceed — but flag it to the user as a gap. - -### Step 2: Query Architectural Decisions - -List all files in `docs/agent/decisions/` and identify which ADRs are relevant to the -current task. If this directory does not exist, skip to Step 3. - -**Relevance matching rules:** - -- Match by filename keywords (e.g., task involves "auth" → read `0002-jwt-auth.md`) -- Match by YAML `tags` in ADR frontmatter if present -- When uncertain, read the ADR — false positives cost less than missed constraints - -**For each relevant ADR, extract and internalize:** - -- `Consequences & Constraints` section → treat as hard rules -- `Actionable Rules` section → treat as implementation requirements -- `Status` field → only `accepted` or `active` ADRs are binding - -See `references/adr-template.md` for the expected ADR structure. - -### Step 3: Check Active Plans - -List files in `docs/agent/plans/` and identify any plan related to the current task. -If this directory does not exist, skip to Step 4. - -- Read the active plan to determine which step is currently being executed -- Do not skip steps unless the user explicitly instructs it -- If no plan exists for the task, proceed but note the absence - -### Step 4: Acknowledge Reading - -Begin the first response to any technical task with a brief acknowledgment: - -``` -I have reviewed: -- `CURRENT_STATE.md`: [one-line summary] -- `decisions/XXXX-name.md`: [relevant constraint noted] -- `plans/active-plan.md`: [current step] - -Proceeding with [task description]... -``` - -If no docs exist yet, state: - -``` -No architecture docs found in docs/agent/. Proceeding without ADR constraints. -Consider scaffolding the agent memory structure if this project needs architectural governance. -``` - -## Conflict Resolution Protocol - -When a user request contradicts an ADR rule: - -1. **STOP** — do not write or propose conflicting code -2. **Quote** the specific rule from the decision file, including the file path -3. **Inform** the user of the conflict clearly: - -``` -⚠️ Conflict detected with `docs/agent/decisions/XXXX-name.md`: - -Rule: "[exact quoted rule]" - -Your request to [description] would violate this constraint. - -Options: - A) Modify the approach to comply with the ADR - B) Update the ADR to allow this exception (I can draft the amendment) - C) Proceed with an explicit architectural exception (will be logged) -``` - -4. **Wait** for the user's decision before proceeding - -## Directory Structure Expected - -``` -docs/agent/ -├── CURRENT_STATE.md # Active milestone, blockers, context -├── decisions/ # Architecture Decision Records -│ ├── 0001-example.md -│ ├── 0002-example.md -│ └── ... -├── plans/ # Active implementation plans -│ ├── active-plan.md -│ └── ... -└── logs/ # Session logs (optional) -``` - -## Graceful Degradation - -Handle missing documentation gracefully: - -| Condition | Action | -|---|---| -| `docs/agent/` missing entirely | Proceed without constraints; suggest scaffolding | -| `CURRENT_STATE.md` missing | Warn user, continue to decisions check | -| `decisions/` empty | Note absence, proceed without ADR constraints | -| `plans/` empty | Proceed without plan context | -| ADR has no `Status` field | Treat as `accepted` (binding) by default | - -## Integration with Existing Workflows - -This protocol runs **before** the existing TradingAgents flows: - -- Before the Agent Flow (analysts → debate → trader → risk) -- Before the Scanner Flow (scanners → deep dive → synthesis) -- Before any CLI changes, config modifications, or test additions - -## Additional Resources - -### Reference Files - -- **`references/adr-template.md`** — Standard ADR template for creating new decisions -- **`references/reading-checklist.md`** — Quick-reference checklist for the reading sequence diff --git a/.claude/skills/architecture-coordinator/references/adr-template.md b/.claude/skills/architecture-coordinator/references/adr-template.md deleted file mode 100644 index 7f317897..00000000 --- a/.claude/skills/architecture-coordinator/references/adr-template.md +++ /dev/null @@ -1,92 +0,0 @@ -# ADR Template - -Architecture Decision Records follow this structure. Use this template when creating -new decisions in `docs/agent/decisions/`. - -## Filename Convention - -``` -NNNN-short-descriptive-name.md -``` - -- `NNNN` — zero-padded sequential number (0001, 0002, ...) -- Use lowercase kebab-case for the name portion - -## Template - -```markdown ---- -title: "Short Decision Title" -status: proposed | accepted | deprecated | superseded -date: YYYY-MM-DD -tags: [relevant, keywords, for, matching] -superseded_by: NNNN-new-decision.md # only if status is superseded ---- - -# NNNN — Short Decision Title - -## Context - -Describe the problem, forces at play, and why a decision is needed. -Include relevant technical constraints, business requirements, and -any alternatives considered. - -## Decision - -State the decision clearly and concisely. Use active voice. - -Example: "Use JWT tokens for API authentication with RS256 signing." - -## Consequences & Constraints - -List the binding rules that follow from this decision. These are -treated as **absolute laws** by the Architecture-First Reading Protocol. - -- **MUST**: [mandatory requirement] -- **MUST NOT**: [explicit prohibition] -- **SHOULD**: [strong recommendation] - -Example: -- MUST use RS256 algorithm for all JWT signing -- MUST NOT store tokens in localStorage -- SHOULD rotate signing keys every 90 days - -## Actionable Rules - -Concrete implementation requirements derived from the decision: - -1. [Specific code/config requirement] -2. [Specific code/config requirement] -3. [Specific code/config requirement] - -## Alternatives Considered - -| Alternative | Reason Rejected | -|---|---| -| Option A | [why not chosen] | -| Option B | [why not chosen] | - -## References - -- [Link or file reference] -- [Related ADR: NNNN-related.md] -``` - -## Status Lifecycle - -``` -proposed → accepted → [deprecated | superseded] -``` - -- **proposed** — Under discussion, not yet binding -- **accepted** — Active and binding; all code must comply -- **deprecated** — No longer relevant; may be ignored -- **superseded** — Replaced by another ADR (link via `superseded_by`) - -## Best Practices - -- Keep decisions focused — one decision per file -- Write constraints as testable statements where possible -- Tag decisions with module/domain keywords for easy matching -- Reference related decisions to build a decision graph -- Date all decisions for historical context diff --git a/.claude/skills/architecture-coordinator/references/reading-checklist.md b/.claude/skills/architecture-coordinator/references/reading-checklist.md deleted file mode 100644 index dc6af768..00000000 --- a/.claude/skills/architecture-coordinator/references/reading-checklist.md +++ /dev/null @@ -1,84 +0,0 @@ -# Architecture Reading Checklist - -Quick-reference checklist for the mandatory reading sequence. -Execute before every technical task. - -## Pre-Flight Checklist - -``` -[ ] 1. Read docs/agent/CURRENT_STATE.md - → Note active milestone - → Note blockers - → Note recent context changes - -[ ] 2. List docs/agent/decisions/*.md - → Identify ADRs relevant to current task - → For each relevant ADR: - [ ] Read Consequences & Constraints - [ ] Read Actionable Rules - [ ] Verify status is accepted/active - [ ] Note any hard prohibitions (MUST NOT) - -[ ] 3. List docs/agent/plans/*.md - → Find active plan for current task - → Identify current step in plan - → Do not skip steps without user approval - -[ ] 4. Acknowledge in response - → List reviewed files - → Summarize relevant constraints - → State intended approach -``` - -## Quick Relevance Matching - -To find relevant ADRs efficiently: - -1. **Extract keywords** from the task description -2. **Match against filenames** in `docs/agent/decisions/` -3. **Check YAML tags** in ADR frontmatter -4. **When in doubt, read it** — a false positive is cheaper than a missed constraint - -### Common Keyword → ADR Mapping Examples - -| Task Keywords | Likely ADR Topics | -|---|---| -| auth, login, token, session | Authentication, authorization | -| database, schema, migration | Data layer, ORM, storage | -| API, endpoint, route | API design, versioning | -| deploy, CI/CD, pipeline | Infrastructure, deployment | -| LLM, model, provider | LLM configuration, vendor routing | -| agent, graph, workflow | Agent architecture, LangGraph | -| config, env, settings | Configuration management | -| test, coverage, fixture | Testing strategy | - -## Conflict Response Template - -When a conflict is detected, use this template: - -``` -⚠️ Conflict detected with `docs/agent/decisions/XXXX-name.md`: - -Rule: "[exact quoted rule from Consequences & Constraints or Actionable Rules]" - -Your request to [brief description of the conflicting action] would violate this constraint. - -Options: - A) Modify the approach to comply with the ADR - B) Update the ADR to allow this exception (I can draft the amendment) - C) Proceed with an explicit architectural exception (will be logged) - -Which option do you prefer? -``` - -## Graceful Degradation Quick Reference - -| Missing Resource | Action | -|---|---| -| Entire `docs/agent/` | Proceed; suggest scaffolding the directory structure | -| `CURRENT_STATE.md` only | Warn, continue to decisions | -| `decisions/` empty | Note absence, proceed freely | -| `plans/` empty | Proceed without plan context | -| ADR missing `Status` | Default to `accepted` (binding) | -| ADR status `proposed` | Informational only, not binding | -| ADR status `deprecated` | Ignore, not binding | diff --git a/.claude/skills/memory-builder/SKILL.md b/.claude/skills/memory-builder/SKILL.md new file mode 100644 index 00000000..c7c9883e --- /dev/null +++ b/.claude/skills/memory-builder/SKILL.md @@ -0,0 +1,320 @@ +--- +name: Memory Builder +description: > + Extracts structured repository knowledge from source code, git history, ADRs, and + conversations, then writes it into the layered memory system under docs/agent/. + Use this skill when asked to "build memory", "update memory", "extract knowledge", + "refresh context files", "rebuild repository docs", "generate memory", "update context", + or any variation of rebuilding or refreshing the project's documentation context. + Also use when a significant code change has landed and the docs/agent/ files may be stale. +version: 1.0.0 +--- + +# Memory Builder + +Build, update, and audit the project's structured memory system. The output files +are the primary context for all future agent sessions — they must be accurate enough +that an agent with zero prior context can understand the project and make correct decisions. + +## Memory Layout + +``` +docs/agent/ +├── CURRENT_STATE.md # Layer 1: Live State (changes every session) +├── context/ # Layer 2: Structured Knowledge (per milestone) +│ ├── ARCHITECTURE.md # System design, workflows, data flow, pipeline +│ ├── CONVENTIONS.md # Coding patterns, rules, gotchas +│ ├── COMPONENTS.md # File map, extension points, test organization +│ ├── TECH_STACK.md # Dependencies, APIs, providers, versions +│ └── GLOSSARY.md # Term definitions, acronyms, data classes +├── decisions/ # Layer 3: Decisions (append-only ADRs) +│ └── NNN-short-name.md +└── plans/ # Layer 4: Plans (implementation checklists) + └── NNN-plan-name.md +``` + +Layers 1-2 are what this skill builds and maintains. Layers 3-4 are written by +other workflows (architecture-coordinator, planning sessions) and are read-only +inputs for this skill. + +## Operational Modes + +Pick the mode that fits the request: + +### Mode 1: Full Build + +Use when: no context files exist, or the user says "rebuild" / "build memory from scratch". + +1. **Audit existing** — read any current `docs/agent/context/*.md` files. Note what exists. +2. **Discover sources** — dynamically scan the codebase (see Discovery below). +3. **Extract** — populate each context file (see Extraction Rules below). +4. **Cross-reference** — verify no contradictions between files. +5. **Quality gate** — run every check (see Quality Gate below). +6. **Report** — output the build report. + +### Mode 2: Targeted Update + +Use when: specific code changed and the user says "update memory" or similar. + +1. Identify which source files changed (check `git diff`, user description, or recent commits). +2. Map changed files to affected context files using the extraction table below. +3. Read only the affected sections, update them, leave the rest untouched. +4. Update the freshness marker on modified files only. + +### Mode 3: Audit + +Use when: the user says "audit memory" or "verify context files". + +1. For each of the 5 context files, pick 5 factual claims at random. +2. Verify each claim against the actual source code. +3. Report: claim, source file checked, pass/fail, correction if needed. +4. Fix any failures found. + +--- + +## Discovery (Step 1) + +Never hardcode file lists. Discover the codebase dynamically: + +``` +# Find all Python source files +find tradingagents/ cli/ -name "*.py" -type f + +# Find config and metadata +ls pyproject.toml .env.example tradingagents/default_config.py + +# Find all class definitions +grep -rn "^class " tradingagents/ cli/ + +# Find all @tool decorated functions +grep -rn "@tool" tradingagents/ + +# Find state/data classes +grep -rn "@dataclass" tradingagents/ cli/ +``` + +### High-Signal Files (read these first) + +| File | Why | +|------|-----| +| `tradingagents/default_config.py` | All config keys, defaults, env var pattern | +| `tradingagents/graph/trading_graph.py` | Trading workflow, agent wiring | +| `tradingagents/graph/scanner_graph.py` | Scanner workflow, parallel execution | +| `tradingagents/graph/setup.py` | Agent factory creation, LLM tiers | +| `tradingagents/agents/utils/tool_runner.py` | Inline tool execution loop | +| `cli/main.py` | CLI commands, entry points | +| `pyproject.toml` | Dependencies, versions, Python constraint | +| `docs/agent/decisions/*.md` | Architectural constraints (binding rules) | + +### Source Priority + +When information conflicts, trust in this order: +1. Source code (always wins) +2. ADRs in `docs/agent/decisions/` +3. Test files +4. Git history +5. Config files +6. README / other docs + +--- + +## Extraction Rules (Step 2) + +### CURRENT_STATE.md + +Three sections only. Max 30 lines total. + +```markdown +# Current Milestone +[One paragraph: what's the active focus and next deliverable] + +# Recent Progress +[Bullet list: what shipped recently, merged PRs, key fixes] + +# Active Blockers +[Bullet list: what's stuck or fragile, with brief context] +``` + +Source: `git log --oneline -20`, open PRs, known issues. + +### ARCHITECTURE.md + +System-level design. Someone reading only this file should understand how the +system works end-to-end. + +| Section | What to extract | Source | +|---------|----------------|--------| +| System Description | One paragraph: what the project is, agent count, vendor count, provider count | `setup.py`, `default_config.py` | +| 3-Tier LLM System | Table: tier name, config key, default model, purpose | `default_config.py` | +| LLM Provider Factory | Table: provider, config value, client class | `setup.py` or wherever `get_llm_client` lives | +| Data Vendor Routing | Table: vendor, capabilities, role (primary/fallback) | `dataflows/`, vendor modules | +| Trading Pipeline | ASCII diagram: analysts → debate → trader → risk → judge | `trading_graph.py` | +| Scanner Pipeline | ASCII diagram: parallel scanners → deep dive → synthesis | `scanner_graph.py` | +| Pipeline Bridge | How scanner output feeds into per-ticker analysis | `macro_bridge.py` or pipeline module | +| CLI Architecture | Commands, UI components (Rich, MessageBuffer) | `cli/main.py` | +| Key Source Files | Table: file path, purpose (10-15 files max) | Discovery step | + +### CONVENTIONS.md + +Rules and patterns. Written as imperatives — "Always...", "Never...", "Use...". +Every convention must cite its source file. + +| Section | What to extract | +|---------|----------------| +| Configuration | Env var override pattern, per-tier overrides, `.env` loading | +| Agent Creation | Factory closure pattern `create_X(llm)`, tool binding rules | +| Tool Execution | Trading: `ToolNode` in graph. Scanners: `run_tool_loop()`. Constants: `MAX_TOOL_ROUNDS`, `MIN_REPORT_LENGTH` | +| Vendor Routing | Fail-fast default (ADR 011), `FALLBACK_ALLOWED` whitelist (list all methods), exception types to catch | +| yfinance Gotchas | `top_companies` has ticker as INDEX not column, `Sector.overview` has no perf data, ETF proxies | +| LangGraph State | Parallel writes need reducers, `_last_value` reducer, list all state classes | +| Threading | Rate limiter: never hold lock during sleep/IO, rate limits per vendor | +| Ollama | Never hardcode `localhost:11434`, use configured `base_url` | +| CLI Patterns | Typer commands, Rich UI, `MessageBuffer`, `StatsCallbackHandler` | +| Pipeline Patterns | `MacroBridge`, `ConvictionLevel`, `extract_json()` | +| Testing | pytest commands, markers, mocking patterns (`VENDOR_METHODS` dict patching), env isolation | +| Error Handling | Fail-fast by default, exception hierarchies per vendor, `raise from` chaining | + +### COMPONENTS.md + +Concrete inventory. The reader should be able to find any file or class quickly. + +| Section | What to extract | +|---------|----------------| +| Directory Tree | Run `find` and format as tree. Verify against actual filesystem. | +| Class Inventory | Table: class name, file, purpose. Use `grep "^class "` to discover. | +| Extension Guides | Step-by-step: how to add a new analyst, scanner, vendor, config key, LLM provider | +| CLI Commands | Table: command name, description, entry point | +| Test Organization | Table: test file, type (unit/integration), what it covers, markers | + +### TECH_STACK.md + +Dependencies and external services. All version constraints come from `pyproject.toml`. + +| Section | What to extract | +|---------|----------------| +| Core Dependencies | Table: package, version constraint (from pyproject.toml), purpose | +| External APIs | Table: service, auth env var, rate limit, primary use | +| LLM Providers | Table: provider, config value, client class, example models | +| Python Version | From pyproject.toml `requires-python` | +| Dev Tools | pytest version, conda, etc. | + +Do NOT include packages that aren't in `pyproject.toml`. Do NOT list aspirational +or unused dependencies. + +### GLOSSARY.md + +Every project-specific term, organized by domain. Each term cites its source file. + +| Domain Section | Terms to include | +|---------------|-----------------| +| Agents & Workflows | Trading Graph, Scanner Graph, Agent Factory, ToolNode, run_tool_loop, Nudge | +| Data Layer | route_to_vendor, VENDOR_METHODS, FALLBACK_ALLOWED, ETF Proxy, etc. | +| Configuration | quick_think, mid_think, deep_think, _env(), _env_int() | +| Vendor-Specific | Exception types (AlphaVantageError, FinnhubError, etc.) | +| State & Data Classes | All @dataclass classes, state types | +| Pipeline | MacroBridge, MacroContext, StockCandidate, TickerResult, ConvictionLevel | +| CLI | MessageBuffer, StatsCallbackHandler, AnalystType, FIXED_AGENTS | +| Constants | All significant constants with actual values and source line | + +--- + +## Quality Gate (Step 3) + +Every context file must pass ALL of these before you're done: + +1. **Accurate** — Every statement is verifiable in the current source code. If you wrote + "17 agents", count them. If you wrote ">=3.10", check pyproject.toml. +2. **Current** — Reflects the latest code on the working branch, not an old snapshot. +3. **Complete** — All 8 subsystems covered: agents, dataflows, graphs, pipeline, CLI, + LLM clients, config, tests. If a subsystem is missing, the gate fails. +4. **Concise** — No information duplicated across context files. Each fact lives in + exactly one file. +5. **Navigable** — A reader can find any specific fact within 2 scrolls or searches. +6. **Quantified** — Constants use actual values from source code (e.g., `MAX_TOOL_ROUNDS=5`), + never vague descriptions ("a maximum number of rounds"). +7. **Cross-referenced** — Every convention cites a source file. Every glossary term + links to where it's defined. + +### Mandatory Verification Steps + +These checks catch the most common errors. Run them before declaring the gate passed. + +1. **Dependency verification**: Parse `pyproject.toml` `[project.dependencies]` and + `[project.optional-dependencies]`. Only list packages that actually appear there. + If a package exists in source imports but not in pyproject.toml, flag it as + "undeclared dependency" — do not silently add it to TECH_STACK. + +2. **Model name verification**: Read `default_config.py` and extract the actual model + identifiers (e.g., `"gpt-4o-mini"`, not guessed names). Cross-check any model names + in ARCHITECTURE.md against what's actually in the config. + +3. **Agent count verification**: Run `grep -rn "def create_" tradingagents/agents/` and + count unique agent factory functions. Use the real count, not an estimate. + +4. **ADR cross-reference verification**: Every ADR cited in context files (e.g., "ADR 011") + must exist in `docs/agent/decisions/`. Run `ls docs/agent/decisions/` and confirm. + +5. **Class existence verification**: Every class listed in COMPONENTS.md must exist in + the codebase. Run `grep -rn "^class ClassName" tradingagents/ cli/` for each one. + +### What NOT to do + +- Do not copy code blocks into docs — reference the file and line instead +- Do not describe aspirational or planned features as current facts +- Do not use stale information from old branches or outdated READMEs +- Do not round numbers — use the exact values from source +- Do not skip CLI or pipeline subsystems (common oversight) +- Do not list dependencies without version constraints from pyproject.toml +- Do not list model names you haven't verified in default_config.py +- Do not include packages from imports that aren't declared in pyproject.toml +- Do not exceed 200 lines per context file — if you're over, split or trim + +## Freshness Markers (Step 4) + +Every context file starts with: +``` + +``` +Set to today's date when creating or updating the file. + +## Build Report (Step 5) + +After completing any mode, output: + +``` +## Memory Build Report + +**Mode**: Build | Update | Audit +**Date**: YYYY-MM-DD + +### Files +| File | Status | Lines | +|------|--------|-------| +| CURRENT_STATE.md | created/updated/unchanged | N | +| context/ARCHITECTURE.md | created/updated/unchanged | N | +| ... | ... | ... | + +### Sources Consulted +- [list of key files read] + +### Quality Gate +| Criterion | Status | +|-----------|--------| +| Accurate | pass/fail | +| Current | pass/fail | +| Complete | pass/fail | +| Concise | pass/fail | +| Navigable | pass/fail | +| Quantified | pass/fail | +| Cross-referenced | pass/fail | + +### Subsystem Coverage +- [x] Agents +- [x] Dataflows +- [x] Graphs +- [x] Pipeline +- [x] CLI +- [x] LLM Clients +- [x] Config +- [x] Tests +``` diff --git a/.claude/skills/memory-reader/SKILL.md b/.claude/skills/memory-reader/SKILL.md new file mode 100644 index 00000000..9b37979f --- /dev/null +++ b/.claude/skills/memory-reader/SKILL.md @@ -0,0 +1,176 @@ +--- +name: Memory Reader +description: > + Reads and applies the project's structured memory system (docs/agent/) at the start + of any technical task. Use this skill at the beginning of every new session, when + starting a new feature or bug fix, when switching to a different part of the codebase, + or when the user says "read memory", "load context", "check decisions", "what's the + current state", "read the ADRs", "what are the conventions", or "catch me up". + This skill acts as a gatekeeper — it ensures all code changes respect established + architecture decisions and current project state. Trigger proactively at session start + before writing any code. +version: 1.0.0 +--- + +# Memory Reader + +Load the project's structured memory before doing any technical work. This skill +ensures you understand the current state, architectural constraints, and coding +conventions before writing or proposing any code changes. + +## When to Read Memory + +Read memory **before** any of these actions: +- Implementing a feature +- Fixing a bug +- Refactoring code +- Adding a new module or agent +- Modifying configuration +- Changing architecture +- Writing or modifying tests + +If `docs/agent/` doesn't exist, skip gracefully and suggest running the memory-builder +skill to set it up. + +## Reading Sequence + +Follow this sequence in order. Each step builds on the previous. + +### Step 1: Current State + +Read `docs/agent/CURRENT_STATE.md` and extract: +- **Active milestone** — what's the current focus? +- **Recent progress** — what just shipped? What PRs merged? +- **Blockers** — what's stuck or fragile? + +This tells you what the team is working on right now and what to be careful around. + +### Step 2: Relevant Architecture Decisions + +List all files in `docs/agent/decisions/` and find ADRs relevant to your current task. + +**How to match relevance:** + +1. Extract keywords from the task (e.g., "add a new vendor" → vendor, fallback, routing) +2. Match against ADR filenames (e.g., `002-data-vendor-fallback.md`, `011-opt-in-vendor-fallback.md`) +3. When uncertain, read the ADR — a false positive costs less than missing a constraint + +**For each relevant ADR, extract:** +- `Consequences & Constraints` — treat as **hard rules** (MUST/MUST NOT) +- `Actionable Rules` — treat as **implementation requirements** +- `Status` — only `accepted` or `active` ADRs are binding + +| Status | Binding? | +|--------|----------| +| `accepted` / `active` | Yes — all code must comply | +| `proposed` | Informational only | +| `deprecated` | Ignore | +| `superseded` | Follow the superseding ADR instead | +| Missing status field | Default to `accepted` (binding) | + +### Step 3: Context Files + +Read the context files relevant to your task. You don't always need all 5 — pick +based on what you're doing: + +| If your task involves... | Read these | +|--------------------------|------------| +| System design, workflows, adding agents | `ARCHITECTURE.md` | +| Writing code, patterns, gotchas | `CONVENTIONS.md` | +| Finding files, classes, extending the system | `COMPONENTS.md` | +| Dependencies, APIs, providers | `TECH_STACK.md` | +| Understanding project terminology | `GLOSSARY.md` | +| Any significant change | All of them | + +Context files live in `docs/agent/context/`. If the directory doesn't exist, note +the absence and proceed without — but flag it to the user. + +### Step 4: Active Plans + +Check `docs/agent/plans/` for any plan related to the current task. + +- If a plan exists, identify the current step and follow it +- Do not skip steps without explicit user approval +- If no plan exists, proceed but note the absence + +### Step 5: Acknowledge + +Start your first response to any technical task with a brief acknowledgment: + +``` +I've reviewed the project memory: +- **State**: [one-line summary of current milestone] +- **ADRs**: [relevant decisions noted, or "none applicable"] +- **Context**: [key conventions or constraints for this task] +- **Plan**: [current step, or "no active plan"] + +Proceeding with [task description]... +``` + +If no docs exist: + +``` +No memory files found in docs/agent/. Proceeding without constraints. +Consider running the memory-builder skill to set up the project memory. +``` + +## Conflict Resolution + +When a user request contradicts an ADR: + +1. **Stop** — do not write conflicting code +2. **Quote** the specific rule, including the file path +3. **Present options**: + +``` +Conflict with `docs/agent/decisions/NNN-name.md`: + +> "[exact quoted rule]" + +Your request to [description] would violate this constraint. + +Options: + A) Modify the approach to comply with the ADR + B) Update the ADR to allow this exception (I can draft the amendment) + C) Proceed with an explicit exception (will be logged) +``` + +4. **Wait** for the user's decision before proceeding + +## Staleness Detection + +While reading, check for signs that the memory is outdated: + +- Freshness markers (``) older than 14 days +- CURRENT_STATE.md mentions milestones that appear completed in git history +- Context files reference files or classes that no longer exist +- Dependency versions in TECH_STACK.md don't match pyproject.toml + +If staleness is detected, warn the user: + +``` +Memory may be stale — [specific issue found]. Consider running the +memory-builder skill in update mode to refresh. +``` + +## Updating State After Work + +After completing a significant task, update `docs/agent/CURRENT_STATE.md`: + +- Add completed work to "Recent Progress" +- Remove resolved items from "Active Blockers" +- Update the milestone summary if it changed + +This keeps the memory fresh for the next session. Only update CURRENT_STATE.md — +the other context files are updated via the memory-builder skill. + +## Graceful Degradation + +| Missing Resource | Action | +|------------------|--------| +| `docs/agent/` entirely | Proceed without constraints; suggest scaffolding | +| `CURRENT_STATE.md` only | Warn, continue to decisions | +| `decisions/` empty | Note absence, proceed freely | +| `context/` empty | Proceed; suggest running memory-builder | +| `plans/` empty | Proceed without plan context | +| Individual context file | Note which is missing, use what's available | diff --git a/.gitignore b/.gitignore index afb65718..281b183f 100644 --- a/.gitignore +++ b/.gitignore @@ -221,7 +221,6 @@ __marimo__/ # Scan results and execution plans (generated artifacts) results/ plans/ -!docs/agent/plans/ # Backup files *.backup diff --git a/docs/agent/CURRENT_STATE.md b/docs/agent/CURRENT_STATE.md index c6100675..14bd64cd 100644 --- a/docs/agent/CURRENT_STATE.md +++ b/docs/agent/CURRENT_STATE.md @@ -1,28 +1,18 @@ # Current Milestone -Pre-existing test failures fixed (12 across 5 files). PR #16 (Finnhub) merged. Next: opt-in vendor fallback (ADR 011), Macro Synthesis JSON robustness, `pipeline` CLI command. +Opt-in vendor fallback (ADR 011) merged (PR #18). Pipeline CLI command implemented. Memory system v2 being built. Next: PR #20 review (Copilot memory rebuild), structured context files under `docs/agent/context/`. # Recent Progress -- End-to-end scanner pipeline operational (`python -m cli.main scan --date YYYY-MM-DD`) -- All 53 tests passing (14 original + 9 scanner fallback + 15 env override + 15 industry deep dive) -- Environment variable config overrides merged (PR #9) -- Thread-safe rate limiter for Alpha Vantage implemented -- Vendor fallback (AV -> yfinance) broadened to catch `AlphaVantageError`, `ConnectionError`, `TimeoutError` -- **PR #13 merged**: Industry Deep Dive quality fixed — enriched industry data (price returns), explicit sector routing via `_extract_top_sectors()`, tool-call nudge in `run_tool_loop` -- Finnhub integrated as third vendor: insider transactions (primary), earnings calendar (new), economic calendar (new) -- ADR 010 written documenting Finnhub vendor decision and paid-tier constraints -- Technical indicators confirmed local-only (stockstats via yfinance OHLCV) — no AV dependency, zero effort to switch -- Finnhub free-tier evaluation complete: 27/41 live tests pass, paid-tier endpoints identified and skipped -- **12 pre-existing test failures fixed** across 5 files: `test_config_wiring.py`, `test_env_override.py`, `test_scanner_comprehensive.py`, `test_scanner_fallback.py`, `test_scanner_graph.py` — root causes: `callable()` wrong for LangChain tools, env var leak via `load_dotenv()` on reload, missing `@pytest.mark.integration` on LLM tests, stale output-file assertions. Full offline suite: 388 passed, 0 failures. - -# Planned Next - -- **Opt-in vendor fallback (ADR 011)** — fail-fast by default, fallback only for fungible data (OHLCV, indices, sector/industry perf, market movers). Plan: `docs/agent/plans/011-opt-in-vendor-fallback.md` -- Macro Synthesis JSON parsing fragile — DeepSeek R1 sometimes wraps output in markdown code blocks; `json.loads()` in CLI may fail (branch `feat/macro-json-robust-parsing` exists with `extract_json()` utility) -- `pipeline` CLI command (scan -> filter -> per-ticker deep dive) not yet implemented +- **PR #18 merged**: Fail-fast vendor routing — `FALLBACK_ALLOWED` whitelist for 5 fungible-data methods only. ADR 011 written, ADR 002 superseded. +- `pipeline` CLI command implemented — scan JSON → filter by conviction → per-ticker deep dive via `MacroBridge` +- `extract_json()` utility in `agents/utils/json_utils.py` handles DeepSeek R1 `` blocks and markdown fences +- All 3 `@dataclass` types defined: `MacroContext`, `StockCandidate`, `TickerResult` in `pipeline/macro_bridge.py` +- 12 pre-existing test failures fixed across 5 files (PR merged to main) +- Memory builder and reader skills created in `.claude/skills/` +- Structured context files generated under `docs/agent/context/` (ARCHITECTURE, CONVENTIONS, COMPONENTS, TECH_STACK, GLOSSARY) # Active Blockers -- Macro Synthesis JSON parsing fragile — DeepSeek R1 sometimes wraps output in markdown code blocks; `json.loads()` in CLI may fail -- `pipeline` CLI command (scan -> filter -> per-ticker deep dive) not yet implemented +- PR #20 (Copilot memory rebuild) is open/draft — needs review for aspirational deps and model name accuracy before merge +- Some scanner integration tests lack `@pytest.mark.integration` marker despite making live network calls diff --git a/docs/agent/context/ARCHITECTURE.md b/docs/agent/context/ARCHITECTURE.md new file mode 100644 index 00000000..350ee1a7 --- /dev/null +++ b/docs/agent/context/ARCHITECTURE.md @@ -0,0 +1,105 @@ + + +# Architecture + +TradingAgents v0.2.1 is a multi-agent LLM framework using LangGraph. It has 17 agent factory functions, 3 data vendors (yfinance, Alpha Vantage, Finnhub), and 6 LLM providers (OpenAI, Anthropic, Google, xAI, OpenRouter, Ollama). + +## 3-Tier LLM System + +| Tier | Config Key | Default Model | Purpose | +|------|-----------|---------------|---------| +| Quick | `quick_think_llm` | `gpt-5-mini` | Analysts, scanners — fast responses | +| Mid | `mid_think_llm` | `None` (falls back to quick) | Bull/bear researchers, trader, deep dive | +| Deep | `deep_think_llm` | `gpt-5.2` | Research manager, risk manager, macro synthesis | + +Each tier has optional `_{tier}_llm_provider` and `_{tier}_backend_url` overrides. All fall back to top-level `llm_provider` (`"openai"`) and `backend_url` (`"https://api.openai.com/v1"`). + +Source: `tradingagents/default_config.py` + +## LLM Provider Factory + +| Provider | Config Value | Client | Notes | +|----------|-------------|--------|-------| +| OpenAI | `"openai"` | `ChatOpenAI` | `openai_reasoning_effort` supported | +| Anthropic | `"anthropic"` | `ChatAnthropic` | — | +| Google | `"google"` | `ChatGoogleGenerativeAI` | `google_thinking_level` supported | +| xAI | `"xai"` | `ChatOpenAI` (OpenAI-compat) | `reasoning_effort` supported | +| OpenRouter | `"openrouter"` | `ChatOpenAI` (OpenAI-compat) | `reasoning_effort` supported | +| Ollama | `"ollama"` | `ChatOpenAI` (OpenAI-compat) | Uses configured `base_url`, never hardcode localhost | + +Source: `tradingagents/llm_clients/` + +## Data Vendor Routing + +| Vendor | Role | Capabilities | +|--------|------|-------------| +| yfinance | Primary (free) | OHLCV, fundamentals, news, screener, sector/industry, indices | +| Alpha Vantage | Fallback | OHLCV, fundamentals, news, sector ETF proxies, market movers | +| Finnhub | Specialized | Insider transactions (primary), earnings calendar, economic calendar | + +Routing: 2-level dispatch — category-level (`data_vendors` config) + tool-level (`tool_vendors` config). Fail-fast by default; only 5 methods in `FALLBACK_ALLOWED` get cross-vendor fallback (ADR 011). + +Source: `tradingagents/dataflows/interface.py` + +## Trading Pipeline + +``` +START ──┬── Market Analyst (quick) ── tools_market ──┐ + ├── Social Analyst (quick) ── tools_social ──┤ + ├── News Analyst (quick) ── tools_news ───────┼── Bull Researcher (mid) ⇄ Bear Researcher (mid) + └── Fundamentals Analyst (quick) ── tools_fund─┘ │ (max_debate_rounds) + Research Manager (deep) + │ + Trader (mid) + │ + Aggressive ⇄ Neutral ⇄ Conservative (quick) + (max_risk_discuss_rounds) + │ + Risk Judge (deep) +``` + +Analysts run in parallel → investment debate → trading plan → risk debate → final decision. + +Source: `tradingagents/graph/trading_graph.py`, `tradingagents/graph/setup.py` + +## Scanner Pipeline + +``` +START ──┬── Geopolitical Scanner (quick) ──┐ + ├── Market Movers Scanner (quick) ──┼── Industry Deep Dive (mid) ── Macro Synthesis (deep) ── END + └── Sector Scanner (quick) ─────────┘ +``` + +Phase 1: 3 scanners run in parallel. Phase 2: Deep dive cross-references all outputs, calls `get_industry_performance` per sector. Phase 3: Macro synthesis produces top-10 watchlist as JSON. + +Source: `tradingagents/graph/scanner_graph.py`, `tradingagents/graph/scanner_setup.py` + +## Pipeline Bridge + +Scanner JSON output → `MacroBridge.load()` → parse into `MacroContext` + `list[StockCandidate]` → `filter_candidates()` by conviction → `run_all_tickers()` (async, `max_concurrent=2`) → per-ticker `TradingAgentsGraph.propagate()` → `save_results()` (per-ticker `.md` + `summary.md` + `results.json`). + +Source: `tradingagents/pipeline/macro_bridge.py` + +## CLI Architecture + +3 Typer commands: `analyze` (interactive per-ticker), `scan` (macro scanner), `pipeline` (scan → filter → deep dive). Rich-based live UI with `MessageBuffer` (deque-backed state manager tracking agent status, reports, tool calls, defined in `cli/main.py`) and `StatsCallbackHandler` (token/timing stats, defined in `cli/stats_handler.py`). 7-step interactive questionnaire in `analyze` for provider/model selection. + +Source: `cli/main.py`, `cli/stats_handler.py` + +## Key Source Files + +| File | Purpose | +|------|---------| +| `tradingagents/default_config.py` | All config keys, defaults, env var override pattern | +| `tradingagents/graph/trading_graph.py` | `TradingAgentsGraph` class, LLM wiring, tool nodes | +| `tradingagents/graph/scanner_graph.py` | `ScannerGraph` class, 3-phase workflow | +| `tradingagents/graph/setup.py` | `GraphSetup` — agent node creation, graph compilation | +| `tradingagents/graph/scanner_setup.py` | `ScannerGraphSetup` — scanner graph compilation | +| `tradingagents/dataflows/interface.py` | `route_to_vendor`, `VENDOR_METHODS`, `FALLBACK_ALLOWED` | +| `tradingagents/agents/utils/tool_runner.py` | `run_tool_loop()`, `MAX_TOOL_ROUNDS=5`, `MIN_REPORT_LENGTH=2000` | +| `tradingagents/agents/utils/agent_states.py` | `AgentState`, `InvestDebateState`, `RiskDebateState` | +| `tradingagents/agents/utils/scanner_states.py` | `ScannerState`, `_last_value` reducer | +| `tradingagents/pipeline/macro_bridge.py` | `MacroBridge`, data classes, pipeline orchestration | +| `tradingagents/agents/utils/json_utils.py` | `extract_json()` — handles DeepSeek R1 markdown wrapping | +| `cli/main.py` | CLI commands, `MessageBuffer`, Rich UI, interactive setup | +| `tradingagents/dataflows/config.py` | `set_config()`, `get_config()`, `initialize_config()` | diff --git a/docs/agent/context/COMPONENTS.md b/docs/agent/context/COMPONENTS.md new file mode 100644 index 00000000..0865261b --- /dev/null +++ b/docs/agent/context/COMPONENTS.md @@ -0,0 +1,188 @@ + + +# Components + +## Directory Tree + +``` +tradingagents/ +├── __init__.py +├── default_config.py # All config keys, defaults, env var overrides +├── agents/ +│ ├── __init__.py +│ ├── analysts/ +│ │ ├── fundamentals_analyst.py # create_fundamentals_analyst(llm) +│ │ ├── market_analyst.py # create_market_analyst(llm) +│ │ ├── news_analyst.py # create_news_analyst(llm) +│ │ └── social_media_analyst.py # create_social_media_analyst(llm) +│ ├── managers/ +│ │ ├── research_manager.py # create_research_manager(llm, memory) +│ │ └── risk_manager.py # create_risk_manager(llm, memory) +│ ├── researchers/ +│ │ ├── bear_researcher.py # create_bear_researcher(llm, memory) +│ │ └── bull_researcher.py # create_bull_researcher(llm, memory) +│ ├── risk_mgmt/ +│ │ ├── aggressive_debator.py # create_aggressive_debator(llm) +│ │ ├── conservative_debator.py # create_conservative_debator(llm) +│ │ └── neutral_debator.py # create_neutral_debator(llm) +│ ├── scanners/ +│ │ ├── __init__.py +│ │ ├── geopolitical_scanner.py # create_geopolitical_scanner(llm) +│ │ ├── market_movers_scanner.py # create_market_movers_scanner(llm) +│ │ ├── sector_scanner.py # create_sector_scanner(llm) +│ │ ├── industry_deep_dive.py # create_industry_deep_dive(llm) +│ │ └── macro_synthesis.py # create_macro_synthesis(llm) +│ ├── trader/ +│ │ └── trader.py # create_trader(llm, memory) +│ └── utils/ +│ ├── agent_states.py # AgentState, InvestDebateState, RiskDebateState +│ ├── agent_utils.py # Tool re-exports, create_msg_delete() +│ ├── core_stock_tools.py # get_stock_data, get_indicators, get_macro_regime +│ ├── fundamental_data_tools.py # get_fundamentals, get_balance_sheet, etc. +│ ├── json_utils.py # extract_json() +│ ├── memory.py # FinancialSituationMemory +│ ├── news_data_tools.py # get_news, get_global_news, get_insider_transactions +│ ├── scanner_states.py # ScannerState, _last_value reducer +│ ├── scanner_tools.py # Scanner @tool definitions (7 tools) +│ ├── technical_indicators_tools.py +│ └── tool_runner.py # run_tool_loop(), MAX_TOOL_ROUNDS, MIN_REPORT_LENGTH +├── dataflows/ +│ ├── __init__.py +│ ├── config.py # set_config(), get_config(), initialize_config() +│ ├── interface.py # route_to_vendor, VENDOR_METHODS, FALLBACK_ALLOWED +│ ├── alpha_vantage.py # Re-export facade +│ ├── alpha_vantage_common.py # Exception hierarchy, rate limiter +│ ├── alpha_vantage_fundamentals.py +│ ├── alpha_vantage_indicator.py +│ ├── alpha_vantage_news.py +│ ├── alpha_vantage_scanner.py +│ ├── alpha_vantage_stock.py +│ ├── finnhub.py +│ ├── finnhub_common.py # Exception hierarchy, rate limiter +│ ├── finnhub_fundamentals.py +│ ├── finnhub_indicators.py +│ ├── finnhub_news.py +│ ├── finnhub_scanner.py +│ ├── finnhub_stock.py +│ ├── macro_regime.py +│ ├── peer_comparison.py +│ ├── stockstats_utils.py +│ ├── ttm_analysis.py +│ ├── utils.py +│ ├── y_finance.py # Core yfinance data functions +│ ├── yfinance_news.py +│ └── yfinance_scanner.py +├── graph/ +│ ├── trading_graph.py # TradingAgentsGraph class +│ ├── scanner_graph.py # ScannerGraph class +│ ├── setup.py # GraphSetup — trading graph builder +│ ├── scanner_setup.py # ScannerGraphSetup — scanner graph builder +│ ├── conditional_logic.py # ConditionalLogic — debate/risk routing +│ ├── propagation.py # Propagator +│ ├── reflection.py # Reflector +│ └── signal_processing.py # SignalProcessor +├── llm_clients/ # Multi-provider LLM factory +│ └── (create_llm_client dispatch) +└── pipeline/ + ├── __init__.py + └── macro_bridge.py # MacroBridge, data classes, pipeline orchestration + +cli/ +└── main.py # Typer app, MessageBuffer, Rich UI, 3 commands +``` + +## Agent Factory Inventory (17 factories + 1 utility) + +| Factory | File | LLM Tier | Extra Params | +|---------|------|----------|-------------| +| `create_fundamentals_analyst` | `agents/analysts/fundamentals_analyst.py` | quick | — | +| `create_market_analyst` | `agents/analysts/market_analyst.py` | quick | — | +| `create_news_analyst` | `agents/analysts/news_analyst.py` | quick | — | +| `create_social_media_analyst` | `agents/analysts/social_media_analyst.py` | quick | — | +| `create_bull_researcher` | `agents/researchers/bull_researcher.py` | mid | `memory` | +| `create_bear_researcher` | `agents/researchers/bear_researcher.py` | mid | `memory` | +| `create_research_manager` | `agents/managers/research_manager.py` | deep | `memory` | +| `create_trader` | `agents/trader/trader.py` | mid | `memory` | +| `create_aggressive_debator` | `agents/risk_mgmt/aggressive_debator.py` | quick | — | +| `create_conservative_debator` | `agents/risk_mgmt/conservative_debator.py` | quick | — | +| `create_neutral_debator` | `agents/risk_mgmt/neutral_debator.py` | quick | — | +| `create_risk_manager` | `agents/managers/risk_manager.py` | deep | `memory` | +| `create_geopolitical_scanner` | `agents/scanners/geopolitical_scanner.py` | quick | — | +| `create_market_movers_scanner` | `agents/scanners/market_movers_scanner.py` | quick | — | +| `create_sector_scanner` | `agents/scanners/sector_scanner.py` | quick | — | +| `create_industry_deep_dive` | `agents/scanners/industry_deep_dive.py` | mid | — | +| `create_macro_synthesis` | `agents/scanners/macro_synthesis.py` | deep | — | +| `create_msg_delete` | `agents/utils/agent_utils.py` | — | No LLM param | + +## Extension Guides + +### Adding a New Analyst +1. Create `tradingagents/agents/analysts/new_analyst.py` with `create_new_analyst(llm)` +2. Add tools to `tradingagents/agents/utils/` and register in `agent_utils.py` +3. Register tool node in `trading_graph.py:_create_tool_nodes()` +4. Add node and edges in `graph/setup.py:setup_graph()` +5. Add conditional routing in `graph/conditional_logic.py` +6. Add to `cli/main.py` `ANALYST_MAPPING` and `REPORT_SECTIONS` + +### Adding a New Scanner +1. Create `tradingagents/agents/scanners/new_scanner.py` with `create_new_scanner(llm)` +2. Export from `agents/scanners/__init__.py` +3. Add to `scanner_graph.py` agents dict +4. Add node and edges in `graph/scanner_setup.py` +5. Add state field with `_last_value` reducer to `scanner_states.py` + +### Adding a New Data Vendor +1. Create `tradingagents/dataflows/newvendor_common.py` (exception hierarchy, rate limiter) +2. Create per-domain modules (`newvendor_stock.py`, `newvendor_scanner.py`, etc.) +3. Add vendor functions to `VENDOR_METHODS` in `interface.py` +4. Add vendor to `VENDOR_LIST` in `interface.py` +5. Add exception types to the catch tuple in `route_to_vendor` +6. Add config category in `default_config.py` `data_vendors` + +### Adding a New LLM Provider +1. Add client creation logic to `tradingagents/llm_clients/` +2. Add provider-specific kwargs handling in `trading_graph.py:_get_provider_kwargs()` and `scanner_graph.py:_get_provider_kwargs()` +3. Add API key to `.env.example` + +### Adding a New Config Key +1. Add to `DEFAULT_CONFIG` dict in `default_config.py` with `_env()` or `_env_int()` override +2. Add to `.env.example` with documentation +3. Update `CLAUDE.md` if it's a frequently-used key + +## CLI Commands + +| Command | Function | Description | +|---------|----------|-------------| +| `analyze` | `run_analysis()` | Interactive per-ticker multi-agent analysis with Rich live UI | +| `scan` | `run_scan(date)` | 3-phase macro scanner, saves 5 report files | +| `pipeline` | `run_pipeline()` | Full pipeline: scan JSON → filter by conviction → per-ticker deep dive | + +## Test Organization + +| Test File | Type | What It Covers | Markers | +|-----------|------|---------------|---------| +| `test_alpha_vantage_exceptions.py` | Mixed | AV exception hierarchy, `_make_api_request` errors | `integration` on HTTP tests | +| `test_alpha_vantage_integration.py` | Unit | Full AV data layer (all mocked) | — | +| `test_alpha_vantage_scanner.py` | Integration | Live AV scanner functions | `integration` | +| `test_config_wiring.py` | Unit | `AgentState` fields, tool exports, debate wiring | — | +| `test_debate_rounds.py` | Unit | `ConditionalLogic` routing at various round counts | — | +| `test_e2e_api_integration.py` | Unit | Vendor routing layer (all mocked) | — | +| `test_env_override.py` | Unit | `TRADINGAGENTS_*` env var overrides | — | +| `test_finnhub_integration.py` | Unit | Full Finnhub data layer (all mocked) | — | +| `test_finnhub_live_integration.py` | Integration | Live Finnhub endpoints | `integration`, `paid_tier` | +| `test_industry_deep_dive.py` | Mixed | `_extract_top_sectors`, nudge mechanism, enriched output | — | +| `test_json_utils.py` | Unit | `extract_json()` — markdown, think blocks, edge cases | — | +| `test_macro_bridge.py` | Unit | Pipeline: parse, filter, render, save | — | +| `test_macro_regime.py` | Mixed | Macro signals, regime classification, report format | `integration` on live test | +| `test_peer_comparison.py` | Mixed | Peer comparison functions | `integration` on live test | +| `test_scanner_comprehensive.py` | Integration | All 5 scanner tools + CLI output naming | — | +| `test_scanner_fallback.py` | Mixed | yfinance perf, AV failure mode, fallback routing | `integration` on some | +| `test_scanner_graph.py` | Unit | `ScannerGraph` import/instantiation, graph compilation | — | +| `test_scanner_mocked.py` | Unit | All yfinance + AV scanner functions (all mocked) | — | +| `test_scanner_routing.py` | Integration | Live routing with AV config | `integration` | +| `test_scanner_tools.py` | Integration | Scanner tool imports + live invocation | — | +| `test_ttm_analysis.py` | Mixed | TTM metrics computation, report format | `integration` on live test | +| `test_vendor_failfast.py` | Unit | ADR 011 fail-fast behavior, error chaining | — | +| `test_yfinance_integration.py` | Unit | Full yfinance data layer (all mocked) | — | + +Pytest markers: `integration` (live API), `paid_tier` (Finnhub paid subscription), `slow` (long-running). Defined in `conftest.py`. diff --git a/docs/agent/context/CONVENTIONS.md b/docs/agent/context/CONVENTIONS.md new file mode 100644 index 00000000..f3d5d180 --- /dev/null +++ b/docs/agent/context/CONVENTIONS.md @@ -0,0 +1,88 @@ + + +# Conventions + +## Configuration + +- Env var override pattern: `TRADINGAGENTS_=value` — empty/unset preserves default. (`default_config.py`) +- Per-tier overrides: each tier has `{tier}_llm_provider` and `{tier}_backend_url`, falling back to top-level `llm_provider` and `backend_url`. (`default_config.py`) +- `load_dotenv()` runs at module level in `default_config.py` — import-order-independent. Check actual env var values when debugging auth. (`default_config.py`) +- `llm_provider` and `backend_url` must always exist at top level — `scanner_graph.py` and `trading_graph.py` use them as fallbacks. (ADR 006) +- `mid_think_llm` defaults to `None`, meaning mid-tier falls back to `quick_think_llm`. (`default_config.py`) + +## Agent Creation + +- Factory pattern: `create_X(llm)` returns a closure `_node(state)`. Some factories take extra params: `create_bull_researcher(llm, memory)`, `create_trader(llm, memory)`. (`tradingagents/agents/`) +- When `bind_tools()` is used, there MUST be a tool execution path — either `ToolNode` in graph or `run_tool_loop()` inline. (ADR 004) + +## Tool Execution + +- Trading graph: analysts use `ToolNode` in the LangGraph graph with conditional routing (`should_continue_X`). (`graph/setup.py`) +- Scanner agents: use `run_tool_loop()` inline — no `ToolNode`, tools execute inside the agent node. (`agents/utils/tool_runner.py`) +- `MAX_TOOL_ROUNDS = 5` — max iterations of tool calling before returning. (`tool_runner.py`) +- `MIN_REPORT_LENGTH = 2000` — if first response is shorter and has no tool calls, a nudge message is appended asking the LLM to call tools. Fires at most once. (`tool_runner.py`) + +## Vendor Routing + +- Fail-fast by default (ADR 011). Only methods in `FALLBACK_ALLOWED` get cross-vendor fallback: + - `get_stock_data` + - `get_market_indices` + - `get_sector_performance` + - `get_market_movers` + - `get_industry_performance` +- Never add news, indicator, or financial-statement tools to `FALLBACK_ALLOWED` — data contracts differ across vendors. (ADR 011) +- Functions inside `route_to_vendor` must RAISE on failure, not embed errors in return values. (`interface.py`) +- Catch `(AlphaVantageError, FinnhubError, ConnectionError, TimeoutError)`, not just `RateLimitError`. (`interface.py`) +- Exception chaining required: `raise RuntimeError(...) from last_error`. (ADR 011) +- 2-level routing: category-level (`data_vendors` config dict) + tool-level override (`tool_vendors` config dict). (`interface.py`) + +## yfinance Gotchas + +- `top_companies` has ticker as the DataFrame INDEX, not a column. Access via `.index`, not a column name. (ADR 003) +- `Sector.overview` has NO performance data. Use ETF proxies (SPDR sector ETFs) for sector performance. (ADR 003) +- Always use `.head(10)` for both download and display in industry performance. (ADR 009) + +## LangGraph State + +- Any state field written by parallel nodes MUST have a reducer (`Annotated[str, reducer_fn]`). (ADR 005) +- `ScannerState` uses `_last_value` reducer (keeps newest value) for all report fields. (`scanner_states.py`) +- State classes: `AgentState` (trading), `InvestDebateState` (debate sub-state), `RiskDebateState` (risk sub-state), `ScannerState` (scanner). (`agent_states.py`, `scanner_states.py`) + +## Threading & Rate Limiting + +- Never hold a lock during `sleep()` or IO. Pattern: release lock, sleep outside, re-acquire. (ADR 007) +- Alpha Vantage: 75 calls/min (premium). (`alpha_vantage_common.py`) +- Finnhub: 60 calls/min (free tier). (`finnhub_common.py`) +- Finnhub paid-tier endpoints (`/stock/candle`, `/financials-reported`, `/indicator`) must never be called on free key. (ADR 010) + +## Ollama + +- Never hardcode `localhost:11434`. Use configured `base_url` from config. (ADR 001) + +## CLI Patterns + +- Typer for command definitions, Rich for live UI. (`cli/main.py`) +- `MessageBuffer` — deque-based singleton tracking agent statuses, reports, tool calls. Fixed agents grouped by team (`FIXED_AGENTS`), analysts selectable. (`cli/main.py`) +- `StatsCallbackHandler` — token and timing statistics for display. (`cli/stats_handler.py`) +- Scan results saved as `{key}.md` files to `results/macro_scan/{scan_date}/`. (`cli/main.py`) + +## Pipeline Patterns + +- `MacroBridge` is the facade class for scan → filter → per-ticker analysis. (`pipeline/macro_bridge.py`) +- `ConvictionLevel = Literal["high", "medium", "low"]`; `CONVICTION_RANK = {"high": 3, "medium": 2, "low": 1}`. (`macro_bridge.py`) +- `extract_json()` handles DeepSeek R1 `` blocks, markdown fences, and raw JSON. (`json_utils.py`) + +## Testing + +- Run tests: `conda activate tradingagents && pytest tests/ -v` +- Skip integration tests: `pytest tests/ -v -m "not integration"` +- Skip paid-tier tests: `pytest tests/ -v -m "not paid_tier"` +- Mocking vendor methods: patch `VENDOR_METHODS` dict entries directly (it stores function refs), not module attributes. (`interface.py`) +- Env isolation: always mock env vars before `importlib.reload()` — `load_dotenv()` leaks real `.env` values otherwise. +- `callable()` returns False on LangChain `@tool` objects — use `hasattr(x, "invoke")` instead. + +## Error Handling + +- Fail-fast by default — no silent fallback unless method is in `FALLBACK_ALLOWED`. (ADR 011) +- Alpha Vantage hierarchy: `AlphaVantageError` → `APIKeyInvalidError`, `RateLimitError`, `ThirdPartyError`, `ThirdPartyTimeoutError`, `ThirdPartyParseError`. (`alpha_vantage_common.py`) +- Finnhub hierarchy: `FinnhubError` → `APIKeyInvalidError`, `RateLimitError`, `ThirdPartyError`, `ThirdPartyTimeoutError`, `ThirdPartyParseError`. (`finnhub_common.py`) diff --git a/docs/agent/context/GLOSSARY.md b/docs/agent/context/GLOSSARY.md new file mode 100644 index 00000000..ee6fc122 --- /dev/null +++ b/docs/agent/context/GLOSSARY.md @@ -0,0 +1,92 @@ + + +# Glossary + +## Agents & Workflows + +| Term | Definition | Source | +|------|-----------|--------| +| Trading Graph | Full per-ticker analysis pipeline: analysts → debate → trader → risk → decision | `graph/trading_graph.py` | +| Scanner Graph | 3-phase macro scanner: parallel scanners → deep dive → synthesis | `graph/scanner_graph.py` | +| Agent Factory | Closure pattern `create_X(llm)` → returns `_node(state)` function | `agents/analysts/*.py`, `agents/scanners/*.py` | +| ToolNode | LangGraph-native tool executor — used in trading graph for analyst tools | `langgraph.prebuilt`, wired in `graph/setup.py` | +| run_tool_loop | Inline tool executor for scanner agents — iterates up to `MAX_TOOL_ROUNDS` | `agents/utils/tool_runner.py` | +| Nudge | If first LLM response is < `MIN_REPORT_LENGTH` chars with no tool calls, a HumanMessage is appended asking LLM to use tools. Fires at most once. | `agents/utils/tool_runner.py` | + +## Data Layer + +| Term | Definition | Source | +|------|-----------|--------| +| route_to_vendor | Central dispatch: resolves vendor for a method, calls it, handles fallback for `FALLBACK_ALLOWED` methods | `dataflows/interface.py` | +| VENDOR_METHODS | Dict mapping method name → vendor → function reference. Direct function refs, not module paths. | `dataflows/interface.py` | +| FALLBACK_ALLOWED | Set of 5 method names that get cross-vendor fallback: `get_stock_data`, `get_market_indices`, `get_sector_performance`, `get_market_movers`, `get_industry_performance` | `dataflows/interface.py` | +| TOOLS_CATEGORIES | Dict mapping category name → `{"description": str, "tools": list}`. 6 categories: `core_stock_apis`, `technical_indicators`, `fundamental_data`, `news_data`, `scanner_data`, `calendar_data` | `dataflows/interface.py` | +| ETF Proxy | SPDR sector ETFs (XLK, XLV, XLF, etc.) used to get sector performance since `Sector.overview` lacks performance data | `yfinance_scanner.py`, `alpha_vantage_scanner.py` | + +## Configuration + +| Term | Definition | Source | +|------|-----------|--------| +| quick_think | Fast-response LLM tier. Default: `gpt-5-mini` via `openai` | `default_config.py` | +| mid_think | Balanced-analysis tier. Default: `None` (falls back to quick_think) | `default_config.py` | +| deep_think | Complex-reasoning tier. Default: `gpt-5.2` via `openai` | `default_config.py` | +| _env() | Helper: reads `TRADINGAGENTS_`, returns default if unset or empty | `default_config.py` | +| _env_int() | Helper: same as `_env()` but coerces to `int` | `default_config.py` | + +## Vendor-Specific + +| Term | Definition | Source | +|------|-----------|--------| +| AlphaVantageError | Base exception for AV failures | `dataflows/alpha_vantage_common.py` | +| FinnhubError | Base exception for Finnhub failures | `dataflows/finnhub_common.py` | +| APIKeyInvalidError | Auth failure (both AV and Finnhub have one) | `*_common.py` | +| RateLimitError | Rate limit exceeded (AV: 75/min, Finnhub: 60/min) | `*_common.py` | +| ThirdPartyError | Generic API error | `*_common.py` | +| ThirdPartyTimeoutError | Request timeout | `*_common.py` | +| ThirdPartyParseError | Response parsing failure | `*_common.py` | +| MSPR | Market Sentiment and Price Return — Finnhub insider transaction metric with no AV/yfinance equivalent | `finnhub_news.py` | + +## State & Data Classes + +| Term | Definition | Source | +|------|-----------|--------| +| AgentState | Trading graph state (extends `MessagesState`). Fields for reports, debate states, trade decision. | `agents/utils/agent_states.py` | +| InvestDebateState | TypedDict sub-state for bull/bear debate. Fields: `bull_history`, `bear_history`, `history`, `current_response`, `judge_decision`, `count`. | `agents/utils/agent_states.py` | +| RiskDebateState | TypedDict sub-state for risk debate. Fields: `aggressive_history`, `conservative_history`, `neutral_history`, `history`, `latest_speaker`, `current_aggressive_response`, `current_conservative_response`, `current_neutral_response`, `judge_decision`, `count`. | `agents/utils/agent_states.py` | +| ScannerState | Scanner graph state (extends `MessagesState`). All report fields use `_last_value` reducer. | `agents/utils/scanner_states.py` | +| _last_value | Reducer function: `def _last_value(existing, new) -> new`. Always keeps the newest value. | `agents/utils/scanner_states.py` | +| FinancialSituationMemory | Memory object for agents that need cross-session recall (bull/bear/trader/judge/risk). | `agents/utils/memory.py` | + +## Pipeline + +| Term | Definition | Source | +|------|-----------|--------| +| MacroBridge | Facade class: load scan JSON → filter candidates → run per-ticker analysis → save results | `pipeline/macro_bridge.py` | +| MacroContext | @dataclass: `economic_cycle`, `central_bank_stance`, `geopolitical_risks`, `key_themes`, `executive_summary`, `risk_factors`, `timeframe`, `region` | `pipeline/macro_bridge.py` | +| StockCandidate | @dataclass: `ticker`, `name`, `sector`, `rationale`, `thesis_angle`, `conviction`, `key_catalysts`, `risks`, `macro_theme` | `pipeline/macro_bridge.py` | +| TickerResult | @dataclass: per-ticker analysis result with all report fields, populated after `propagate()` | `pipeline/macro_bridge.py` | +| ConvictionLevel | `Literal["high", "medium", "low"]` | `pipeline/macro_bridge.py` | +| CONVICTION_RANK | `{"high": 3, "medium": 2, "low": 1}` — used for sorting/filtering | `pipeline/macro_bridge.py` | + +## CLI + +| Term | Definition | Source | +|------|-----------|--------| +| MessageBuffer | Deque-based singleton tracking agent statuses, reports, tool calls for Rich live UI | `cli/main.py` | +| StatsCallbackHandler | Token and timing statistics handler for display | `cli/stats_handler.py` | +| FIXED_AGENTS | Dict grouping non-analyst agents by team: Research Team, Trading Team, Risk Management, Portfolio Management | `cli/main.py` | +| ANALYST_MAPPING | Dict: `"market"` → `"Market Analyst"`, `"social"` → `"Social Analyst"`, etc. | `cli/main.py` | + +## Constants + +| Constant | Value | Source | +|----------|-------|--------| +| MAX_TOOL_ROUNDS | `5` | `agents/utils/tool_runner.py` | +| MIN_REPORT_LENGTH | `2000` | `agents/utils/tool_runner.py` | +| max_debate_rounds | `1` (default) | `default_config.py` | +| max_risk_discuss_rounds | `1` (default) | `default_config.py` | +| max_recur_limit | `100` (default) | `default_config.py` | +| AV _RATE_LIMIT | `75` calls/min | `dataflows/alpha_vantage_common.py` | +| Finnhub _RATE_LIMIT | `60` calls/min | `dataflows/finnhub_common.py` | +| AV API_BASE_URL | `"https://www.alphavantage.co/query"` | `dataflows/alpha_vantage_common.py` | +| Finnhub API_BASE_URL | `"https://finnhub.io/api/v1"` | `dataflows/finnhub_common.py` | diff --git a/docs/agent/context/TECH_STACK.md b/docs/agent/context/TECH_STACK.md new file mode 100644 index 00000000..f99f072c --- /dev/null +++ b/docs/agent/context/TECH_STACK.md @@ -0,0 +1,75 @@ + + +# Tech Stack + +## Python Version + +`>=3.10` (from `pyproject.toml` `requires-python`) + +## Core Dependencies + +All from `pyproject.toml` `[project.dependencies]`: + +| Package | Constraint | Purpose | +|---------|-----------|---------| +| `langchain-core` | `>=0.3.81` | Base LangChain abstractions, messages, tools | +| `langchain-anthropic` | `>=0.3.15` | Anthropic LLM provider | +| `langchain-google-genai` | `>=2.1.5` | Google Gemini LLM provider | +| `langchain-openai` | `>=0.3.23` | OpenAI/xAI/OpenRouter/Ollama LLM provider | +| `langchain-experimental` | `>=0.3.4` | Experimental LangChain features | +| `langgraph` | `>=0.4.8` | Graph-based agent orchestration | +| `yfinance` | `>=0.2.63` | Primary data vendor (stocks, fundamentals, news) | +| `pandas` | `>=2.3.0` | DataFrame operations for financial data | +| `stockstats` | `>=0.6.5` | Technical indicators from OHLCV data | +| `python-dotenv` | `>=1.0.0` | `.env` file loading | +| `typer` | `>=0.21.0` | CLI framework | +| `rich` | `>=14.0.0` | Terminal UI (panels, tables, live display) | +| `requests` | `>=2.32.4` | HTTP client for AV/Finnhub APIs | +| `redis` | `>=6.2.0` | Caching layer | +| `questionary` | `>=2.1.0` | Interactive CLI prompts | +| `backtrader` | `>=1.9.78.123` | Backtesting framework | +| `chainlit` | `>=2.5.5` | Web UI framework | +| `parsel` | `>=1.10.0` | HTML/XML parsing | +| `rank-bm25` | `>=0.2.2` | BM25 text ranking | +| `pytz` | `>=2025.2` | Timezone handling | +| `tqdm` | `>=4.67.1` | Progress bars | +| `typing-extensions` | `>=4.14.0` | Backported typing features | +| `setuptools` | `>=80.9.0` | Package build system | + +## Dev Dependencies + +From `[dependency-groups]`: + +| Package | Constraint | Purpose | +|---------|-----------|---------| +| `pytest` | `>=9.0.2` | Test framework | + +## External APIs + +| Service | Auth Env Var | Rate Limit | Primary Use | +|---------|-------------|-----------|-------------| +| Alpha Vantage | `ALPHA_VANTAGE_API_KEY` | 75/min (premium) | Fallback data vendor | +| Finnhub | `FINNHUB_API_KEY` | 60/min (free) | Insider transactions, calendars | +| OpenAI | `OPENAI_API_KEY` | Per plan | Default LLM provider | +| Anthropic | `ANTHROPIC_API_KEY` | Per plan | LLM provider | +| Google | `GOOGLE_API_KEY` | Per plan | LLM provider (Gemini) | +| xAI | `XAI_API_KEY` | Per plan | LLM provider (Grok) | +| OpenRouter | `OPENROUTER_API_KEY` | Per plan | LLM provider (multi-model) | + +## LLM Provider Support + +| Provider | Config Value | Client Class | Notes | +|----------|-------------|-------------|-------| +| OpenAI | `"openai"` | `ChatOpenAI` | Default. `openai_reasoning_effort` optional. | +| Anthropic | `"anthropic"` | `ChatAnthropic` | — | +| Google | `"google"` | `ChatGoogleGenerativeAI` | `google_thinking_level` optional. | +| xAI | `"xai"` | `ChatOpenAI` | OpenAI-compatible endpoint. | +| OpenRouter | `"openrouter"` | `ChatOpenAI` | OpenAI-compatible endpoint. | +| Ollama | `"ollama"` | `ChatOpenAI` | OpenAI-compatible. Uses configured `base_url`. | + +## Project Metadata + +- Name: `tradingagents` +- Version: `0.2.1` +- Entry point: `tradingagents = cli.main:app` +- Package discovery: `tradingagents*`, `cli*`