charieswei
/
TradingAgents
mirror of https://github.com/TauricResearch/TradingAgents.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
79 Commits 2 Branches 6 Tags 543 MiB
Commit Graph

7 Commits

Author SHA1 Message Date
Youssef Aitousarrah f6943e1615 fix: add noqa comment to prevent linter from removing scanner import 
The scanner import needs # noqa: F401 to prevent linters from
removing it as "unused". The import is required for side effects
(triggering scanner registration).

Without this:
- Pre-commit hook removes the import
- Scanners don't register
- Discovery returns 0 candidates

Fix:
- Added # noqa: F401 comment to scanner import
- Linter will now preserve this import
- Verified 8 scanners properly registered

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-02-05 23:46:27 -08:00
Youssef Aitousarrah 1d52211383 fix: restore missing scanner import causing 0 recommendations 
Critical bugfix: Scanner modules weren't being imported, causing
SCANNER_REGISTRY to remain empty and discovery to return 0 candidates.

Root Cause:
- Import line "from tradingagents.dataflows.discovery import scanners"
  was accidentally removed during concurrent execution refactoring
- Without this import, scanner @register() decorators never execute
- Result: SCANNER_REGISTRY.get_all_scanners() returns empty list

Fix:
- Restored scanner import in discovery_graph.py line 6
- Scanners now properly register on module import
- Verified 8 scanners now registered and working

Impact:
- Before: 0 candidates, 0 recommendations
- After: 60-70 candidates, 15 recommendations (normal operation)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-02-05 23:39:20 -08:00
Youssef Aitousarrah 369f8c444b feat: discovery system code quality improvements and concurrent execution 
Implement comprehensive code quality improvements and performance optimizations
for the discovery pipeline based on code review findings.

## Key Improvements

### 1. Common Utilities (DRY Principle)
- Created `tradingagents/dataflows/discovery/common_utils.py`
- Extracted ticker parsing logic (eliminates 40+ lines of duplication)
- Centralized stopwords list (71 common non-ticker words)
- Added ReDoS protection (100KB text length limit)
- Provides `validate_candidate_structure()` for output validation

### 2. Scanner Output Validation
- Two-layer validation approach:
  - Registration-time: Check scanner class structure
  - Runtime: Validate each candidate dictionary
- Added `scan_with_validation()` wrapper in BaseScanner
- Validates required keys: ticker, source, context, priority
- Graceful error handling with structured logging

### 3. Configuration-Driven Design
- Moved magic numbers to `default_config.py`:
  - `ticker_universe`: Top 20 liquid options tickers
  - `min_volume`: 1000 (options flow threshold)
  - `min_transaction_value`: $25,000 (insider buying filter)
- Fixed hardcoded absolute paths to relative paths
- Improved portability across development environments

### 4. Concurrent Scanner Execution (37% Performance Gain)
- Implemented ThreadPoolExecutor for parallel scanner execution
- Configuration: `scanner_execution.concurrent`, `max_workers`, `timeout_seconds`
- Performance: 42s vs 67s (37% faster with 8 scanners)
- Thread-safe state management (each scanner gets copy)
- Per-scanner timeout with graceful degradation
- Error isolation (one failure doesn't stop others)

### 5. Error Handling Improvements
- Changed bare `except:` to `except Exception:` (avoid catching KeyboardInterrupt)
- Added structured logging with `exc_info=True` and extra fields
- Implemented graceful degradation throughout pipeline

## Files Changed

### Core Implementation
- `tradingagents/__init__.py` (NEW) - Package initialization
- `tradingagents/default_config.py` - Scanner execution config, magic numbers
- `tradingagents/graph/discovery_graph.py` - Concurrent execution logic
- `tradingagents/dataflows/discovery/common_utils.py` (NEW) - Shared utilities
- `tradingagents/dataflows/discovery/scanner_registry.py` - Validation wrapper
- `tradingagents/dataflows/discovery/scanners/*.py` - Use common utilities

### Testing & Documentation
- `tests/test_concurrent_scanners.py` (NEW) - Comprehensive test suite
- `verify_concurrent_execution.py` (NEW) - Performance verification
- `CONCURRENT_EXECUTION.md` (NEW) - Implementation documentation

## Test Results

All tests passing (exit code 0):
- ✅ Concurrent execution: 42s, 66-69 candidates
- ✅ Sequential fallback: 56-67s, 65-68 candidates
- ✅ Timeout handling: Graceful degradation with 1s timeout
- ✅ Error isolation: Individual failures don't cascade

## Performance Impact

- Scanner execution: 37% faster (42s vs 67s)
- Time saved: ~25 seconds per discovery run
- At scale: 4+ minutes saved daily in production
- Same candidate quality (65-69 tickers in both modes)

## Breaking Changes

None. Concurrent execution is opt-in via config flag.
Sequential mode remains available as fallback.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-02-05 23:27:01 -08:00
Youssef Aitousarrah 2376fc74a1 Update 2025-12-11 00:23:28 -08:00
Youssef Aitousarrah ea4ee9176b Update 2025-12-09 23:16:53 -08:00
Youssef Aitousarrah ccc78c694b Update 2025-12-06 15:39:49 -08:00
Youssef Aitousarrah 5cf57e5d97 Update 2025-12-02 20:49:42 -08:00