Commit Graph

276 Commits

Author SHA1 Message Date
github-actions[bot] 4f26536f46 chore: daily discovery 2026-02-20 2026-02-20 13:46:15 +00:00
github-actions[bot] aadb54e776 chore: update performance tracking 2026-02-19 2026-02-19 13:55:01 +00:00
github-actions[bot] 89c23751fa chore: daily discovery 2026-02-19 2026-02-19 13:54:25 +00:00
github-actions[bot] 4355f633b3 chore: update performance tracking 2026-02-18 2026-02-18 13:51:57 +00:00
github-actions[bot] a29cb9ca18 chore: daily discovery 2026-02-18 2026-02-18 13:51:26 +00:00
Youssef Aitousarrah 74e8703d60 Add daily price movement to charts and fix intraday data rendering
- Add daily price movement display with color coding (green/red)
- Add 1D (intraday) and 7D chart options with granular data:
  - 1D: 5-minute interval for detailed intraday view
  - 7D: hourly interval for smooth 7-day chart
- Fix discontinuous chart rendering by plotting against sequential index for intraday data
- Eliminate overnight/weekend gaps in hourly charts
- Add timezone normalization for consistent date handling between daily and intraday data
- Improve fallback logic when data is sparse
- Better handling of yfinance column names (Datetime vs Date)

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
2026-02-17 12:55:12 -08:00
Aitous f4dba2940c
Merge pull request #8 from Aitous/feature/discovery-enhancements
Update
2026-02-17 12:14:19 -08:00
Youssef Aitousarrah 22433d6b7d Update 2026-02-17 12:12:20 -08:00
Aitous f8c6e2afcb
Merge pull request #7 from Aitous/feature/discovery-enhancements
Update
2026-02-17 12:08:22 -08:00
Youssef Aitousarrah fd951be8bc Update 2026-02-17 12:07:07 -08:00
Aitous feccdb77fc
Merge pull request #6 from Aitous/feature/discovery-enhancements
Feature/discovery enhancements
2026-02-17 10:27:57 -08:00
Youssef Aitousarrah 457d650e42 Update 2026-02-17 10:27:13 -08:00
github-actions[bot] 60a4fdd98d chore: update performance tracking 2026-02-17 2026-02-17 13:51:48 +00:00
github-actions[bot] 743ef2c56e chore: daily discovery 2026-02-17 2026-02-17 13:50:40 +00:00
Youssef Aitousarrah aef83cd6e0 Update the price chart 2026-02-16 22:41:54 -08:00
Aitous 3c0a6b427c
Merge pull request #5 from Aitous/feature/discovery-enhancements
Feature/discovery enhancements
2026-02-16 14:18:29 -08:00
Youssef Aitousarrah 6831339b78 Remore unused code and improve the UI 2026-02-16 14:17:43 -08:00
Youssef Aitousarrah 8d3205043e Update 2026-02-16 14:17:41 -08:00
Aitous 080f626a22 Update cron schedule to run one hour earlier 2026-02-16 13:27:01 -08:00
Aitous 7f326b1df8 Update cron schedule to run one hour earlier 2026-02-16 13:25:57 -08:00
Aitous 763c44691e
Merge pull request #4 from Aitous/feature/discovery-enhancements
feat: add performance tracking step to daily discovery workflow
2026-02-16 13:24:57 -08:00
Youssef Aitousarrah 683d42a9e2 feat: add performance tracking step to daily discovery workflow
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 13:24:15 -08:00
github-actions[bot] e3f3c02dc9 chore: daily discovery 2026-02-16 2026-02-16 14:32:42 +00:00
Aitous f535b78b2e Update cron expression in daily-discovery.yml 2026-02-15 14:37:51 -08:00
Aitous bb1c32f3ea Update cron expression to run daily including weekends 2026-02-15 14:36:41 -08:00
github-actions[bot] 18f6907857 chore: daily discovery 2026-02-13 2026-02-13 14:20:20 +00:00
github-actions[bot] 85ec1cb88f chore: daily discovery 2026-02-12 2026-02-12 14:38:41 +00:00
github-actions[bot] 53078c6319 chore: daily discovery 2026-02-12 2026-02-12 14:22:14 +00:00
github-actions[bot] 54f5af2036 chore: daily discovery 2026-02-12 2026-02-12 06:47:07 +00:00
Aitous c4fe114e84
Merge pull request #3 from Aitous/feature/discovery-enhancements
fix: add missing tweepy dependency to pyproject.toml
2026-02-11 22:32:40 -08:00
Youssef Aitousarrah b41fe8c69c fix: add missing tweepy dependency to pyproject.toml
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-11 22:26:51 -08:00
Aitous 9442de7868
Merge pull request #2 from Aitous/feature/discovery-enhancements
Feature/discovery enhancements
2026-02-11 22:12:06 -08:00
Youssef Aitousarrah f4aceef857 feat: add daily discovery workflow, recommendation history, and scanner improvements
- Add GitHub Actions workflow for daily discovery (8:30 AM ET, weekdays)
- Add headless run_daily_discovery.py script for scheduling
- Expand options_flow scanner to use tickers.txt with parallel execution
- Add recommendation history section to Performance page with filters and charts
- Fix strategy name normalization (momentum/Momentum/Momentum-Hype → momentum)
- Fix strategy metrics to count all recs, not just evaluated ones
- Add error handling to Streamlit page rendering

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-11 22:07:02 -08:00
Youssef Aitousarrah ab8d174990 Add recommendations folder so that the UI can display it 5 2026-02-10 22:43:46 -08:00
Youssef Aitousarrah 1ead4d9638 feat: add theme module and fix Streamlit Cloud deployment
- Add tradingagents/ui/theme.py (design system: colors, CSS, Plotly templates)
- Add .streamlit/config.toml for dark theme configuration
- Fix Plotly duplicate keyword args in performance.py and todays_picks.py
- Replace deprecated use_container_width with width="stretch" (Streamlit 1.54+)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-10 22:40:08 -08:00
Youssef Aitousarrah 8ebb42114d Add recommendations folder so that the UI can display it 4 2026-02-10 22:28:52 -08:00
Youssef Aitousarrah 0bc7dda086 Add recommendations folder so that the UI can display it 3 2026-02-10 09:43:54 -08:00
Youssef Aitousarrah 50eee9b945 Add recommendations folder so that the UI can display it 2 2026-02-09 23:53:55 -08:00
Youssef Aitousarrah 8273870e0a Add recommendations folder so that the UI can display it 2026-02-09 23:41:57 -08:00
Youssef Aitousarrah 259705d1d6 Add streamlit app file 2026-02-09 23:28:10 -08:00
Aitous fc1dac665d Added Dev Container Folder 2026-02-09 23:22:02 -08:00
Youssef Aitousarrah cb5ae49501 chore: linter formatting + ML scanner logging, prompt control, ranker reasoning
- Add ML signal scanner results table logging
- Add log_prompts_console config flag for prompt visibility control
- Expand ranker investment thesis to 4-6 sentence structured reasoning
- Linter auto-formatting across modified files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-09 23:04:38 -08:00
Youssef Aitousarrah 43bdd6de11 feat: discovery pipeline enhancements with ML signal scanner
Major additions:
- ML win probability scanner: scans ticker universe using trained
  LightGBM/TabPFN model, surfaces candidates with P(WIN) above threshold
- 30-feature engineering pipeline (20 base + 10 interaction features)
  computed from OHLCV data via stockstats + pandas
- Triple-barrier labeling for training data generation
- Dataset builder and training script with calibration analysis
- Discovery enrichment: confluence scoring, short interest extraction,
  earnings estimates, options signal normalization, quant pre-score
- Configurable prompt logging (log_prompts_console flag)
- Enhanced ranker investment thesis (4-6 sentence reasoning)
- Typed DiscoveryConfig dataclass for all discovery settings
- Console price charts for visual ticker analysis

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-09 22:53:42 -08:00
Youssef Aitousarrah 1d78271ef4 chore: ignore .worktrees directory
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-06 09:25:14 -08:00
Youssef Aitousarrah 6f202f88f2 docs: add volume analysis enhancements design document
Complete design for transforming get_unusual_volume into sophisticated
multi-signal analysis tool with:
- Volume pattern analysis (accumulation, compression, distribution)
- Sector-relative comparison (percentile ranking vs peers)
- Price-volume divergence detection (bullish/bearish signals)

Includes architecture, implementation details, testing strategy,
and performance considerations. Estimated 30-40% signal quality
improvement with 2-3x execution time trade-off.

Phased implementation approach:
Phase 1: Pattern analysis (3-4h)
Phase 2: Divergence detection (4-5h)
Phase 3: Sector comparison (5-6h)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-06 08:38:06 -08:00
Youssef Aitousarrah f1178b4a57 refactor: organize discovery config into dedicated filter/enrichment sections
- Created nested "filters" section for all filter-stage settings
  (min_average_volume, same-day movers, recent movers, etc.)
- Created nested "enrichment" section for batch news settings
- Updated CandidateFilter to read from new nested structure
- Added backward compatibility fallback for old flat config
- Improved config organization and clarity

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-06 08:22:39 -08:00
Youssef Aitousarrah 41e91e72d1 fix: load scanners in __init__ to survive linter auto-fix
Final fix for scanner registration issue. Previous attempts
to add scanner import at module level were removed by the
pre-commit hook's ruff --fix auto-formatter.

Solution:
- Import scanners inside DiscoveryGraph.__init__() method
- Use the import (assign to _) so it's not "unused"
- Linter won't remove imports that are actually used

This ensures scanners always load when DiscoveryGraph is instantiated.

Verified: 8 scanners now properly registered

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-05 23:47:26 -08:00
Youssef Aitousarrah f6943e1615 fix: add noqa comment to prevent linter from removing scanner import
The scanner import needs # noqa: F401 to prevent linters from
removing it as "unused". The import is required for side effects
(triggering scanner registration).

Without this:
- Pre-commit hook removes the import
- Scanners don't register
- Discovery returns 0 candidates

Fix:
- Added # noqa: F401 comment to scanner import
- Linter will now preserve this import
- Verified 8 scanners properly registered

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-05 23:46:27 -08:00
Youssef Aitousarrah 1d52211383 fix: restore missing scanner import causing 0 recommendations
Critical bugfix: Scanner modules weren't being imported, causing
SCANNER_REGISTRY to remain empty and discovery to return 0 candidates.

Root Cause:
- Import line "from tradingagents.dataflows.discovery import scanners"
  was accidentally removed during concurrent execution refactoring
- Without this import, scanner @register() decorators never execute
- Result: SCANNER_REGISTRY.get_all_scanners() returns empty list

Fix:
- Restored scanner import in discovery_graph.py line 6
- Scanners now properly register on module import
- Verified 8 scanners now registered and working

Impact:
- Before: 0 candidates, 0 recommendations
- After: 60-70 candidates, 15 recommendations (normal operation)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-05 23:39:20 -08:00
Youssef Aitousarrah 369f8c444b feat: discovery system code quality improvements and concurrent execution
Implement comprehensive code quality improvements and performance optimizations
for the discovery pipeline based on code review findings.

## Key Improvements

### 1. Common Utilities (DRY Principle)
- Created `tradingagents/dataflows/discovery/common_utils.py`
- Extracted ticker parsing logic (eliminates 40+ lines of duplication)
- Centralized stopwords list (71 common non-ticker words)
- Added ReDoS protection (100KB text length limit)
- Provides `validate_candidate_structure()` for output validation

### 2. Scanner Output Validation
- Two-layer validation approach:
  - Registration-time: Check scanner class structure
  - Runtime: Validate each candidate dictionary
- Added `scan_with_validation()` wrapper in BaseScanner
- Validates required keys: ticker, source, context, priority
- Graceful error handling with structured logging

### 3. Configuration-Driven Design
- Moved magic numbers to `default_config.py`:
  - `ticker_universe`: Top 20 liquid options tickers
  - `min_volume`: 1000 (options flow threshold)
  - `min_transaction_value`: $25,000 (insider buying filter)
- Fixed hardcoded absolute paths to relative paths
- Improved portability across development environments

### 4. Concurrent Scanner Execution (37% Performance Gain)
- Implemented ThreadPoolExecutor for parallel scanner execution
- Configuration: `scanner_execution.concurrent`, `max_workers`, `timeout_seconds`
- Performance: 42s vs 67s (37% faster with 8 scanners)
- Thread-safe state management (each scanner gets copy)
- Per-scanner timeout with graceful degradation
- Error isolation (one failure doesn't stop others)

### 5. Error Handling Improvements
- Changed bare `except:` to `except Exception:` (avoid catching KeyboardInterrupt)
- Added structured logging with `exc_info=True` and extra fields
- Implemented graceful degradation throughout pipeline

## Files Changed

### Core Implementation
- `tradingagents/__init__.py` (NEW) - Package initialization
- `tradingagents/default_config.py` - Scanner execution config, magic numbers
- `tradingagents/graph/discovery_graph.py` - Concurrent execution logic
- `tradingagents/dataflows/discovery/common_utils.py` (NEW) - Shared utilities
- `tradingagents/dataflows/discovery/scanner_registry.py` - Validation wrapper
- `tradingagents/dataflows/discovery/scanners/*.py` - Use common utilities

### Testing & Documentation
- `tests/test_concurrent_scanners.py` (NEW) - Comprehensive test suite
- `verify_concurrent_execution.py` (NEW) - Performance verification
- `CONCURRENT_EXECUTION.md` (NEW) - Implementation documentation

## Test Results

All tests passing (exit code 0):
-  Concurrent execution: 42s, 66-69 candidates
-  Sequential fallback: 56-67s, 65-68 candidates
-  Timeout handling: Graceful degradation with 1s timeout
-  Error isolation: Individual failures don't cascade

## Performance Impact

- Scanner execution: 37% faster (42s vs 67s)
- Time saved: ~25 seconds per discovery run
- At scale: 4+ minutes saved daily in production
- Same candidate quality (65-69 tickers in both modes)

## Breaking Changes

None. Concurrent execution is opt-in via config flag.
Sequential mode remains available as fallback.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-05 23:27:01 -08:00