diff --git a/docs/iterations/LEARNINGS.md b/docs/iterations/LEARNINGS.md index 8fd32c43..b5d7da66 100644 --- a/docs/iterations/LEARNINGS.md +++ b/docs/iterations/LEARNINGS.md @@ -1,20 +1,20 @@ # Learnings Index -**Last analyzed run:** 2026-04-11 +**Last analyzed run:** 2026-04-12 | Domain | File | Last Updated | One-line Summary | |--------|------|--------------|-----------------| -| options_flow | scanners/options_flow.md | 2026-04-11 | 46% 7d win rate; signal decays rapidly past 1 week | -| insider_buying | scanners/insider_buying.md | 2026-04-11 | -2.05% 30d avg; raised min-txn to $100K to reduce noise | +| options_flow | scanners/options_flow.md | 2026-04-12 | Premium filter confirmed applied; CSCO cross-scanner confluence detected; 45.6% 7d win rate | +| insider_buying | scanners/insider_buying.md | 2026-04-12 | Staleness pattern (HMH 4 consecutive days); 38.1% 1d, 46.4% 7d win rates — worst volume-to-quality ratio | +| minervini | scanners/minervini.md | 2026-04-12 | Best performer: 100% 1d win rate (n=3), +3.68% avg; 7 candidates in Apr 6-12 week | +| analyst_upgrades | scanners/analyst_upgrades.md | 2026-04-12 | 50% 7d win rate (breakeven); cross-scanner confluence with options_flow is positive signal | +| earnings_calendar | scanners/earnings_calendar.md | 2026-04-12 | Appears as earnings_play; 38.1% 1d, 37.7% 7d — poor; best setups require high short interest | +| pipeline/scoring | pipeline/scoring.md | 2026-04-12 | stats summary now surfaces worst performers; news_catalyst 0% 7d, social_hype 14.3% 7d — worst strategies | | volume_accumulation | scanners/volume_accumulation.md | — | No data yet | -| reddit_dd | scanners/reddit_dd.md | 2026-04-11 | Only positive strategy: +0.94% 30d avg, 55% 30d win rate | -| reddit_trending | scanners/reddit_trending.md | 2026-04-11 | -10.64% 30d avg; restricted to HIGH priority (>=50 mentions) | -| semantic_news | scanners/semantic_news.md | 2026-04-11 | -17.5% 30d avg; restricted to CRITICAL catalysts only | +| reddit_dd | scanners/reddit_dd.md | — | No data yet | +| reddit_trending | scanners/reddit_trending.md | — | No data yet | +| semantic_news | scanners/semantic_news.md | — | No data yet | | market_movers | scanners/market_movers.md | — | No data yet | -| earnings_calendar | scanners/earnings_calendar.md | — | No data yet | -| analyst_upgrades | scanners/analyst_upgrades.md | — | No data yet | | technical_breakout | scanners/technical_breakout.md | — | No data yet | | sector_rotation | scanners/sector_rotation.md | — | No data yet | | ml_signal | scanners/ml_signal.md | — | No data yet | -| minervini | scanners/minervini.md | 2026-04-11 | 100% 1d win rate (4 pts); Stage 2 filter effective in downturn | -| pipeline/scoring | pipeline/scoring.md | 2026-04-11 | Strategy identity predicts outcomes better than final_score | diff --git a/docs/iterations/pipeline/scoring.md b/docs/iterations/pipeline/scoring.md index 31127cf2..2a1b66ff 100644 --- a/docs/iterations/pipeline/scoring.md +++ b/docs/iterations/pipeline/scoring.md @@ -4,29 +4,25 @@ LLM assigns a final_score (0-100) and confidence (1-10) to each candidate. Score and confidence are correlated but not identical — a speculative setup can score 80 with confidence 6. The ranker uses final_score as primary sort key. - -P&L data provides first evidence on score vs. outcome relationship: overall 30d -win rate is only 33.8% despite most recommendations having final_score >= 65. -This suggests the LLM is systematically overconfident — scores in the 65-85 range -do not reliably predict positive outcomes. Strategy identity (which scanner sourced -the candidate) is a stronger predictor than score within that strategy. +No evidence yet on whether confidence or score is a better predictor of outcomes. ## Evidence Log -### 2026-04-11 — P&L review -- 608 total recommendations, 30d win rate 33.8%, avg 30d return -2.9%. -- Score distribution in sample files: most recs scored 65-92. Win rate at 30d is - 33.8% overall — scores in this range are not predictive of positive outcomes. -- Strategy is a stronger predictor than score: social_dd (55% 30d win rate) vs. - social_hype (15.4% 30d win rate) despite similar score distributions. -- Confidence calibration: scores of 85+ with confidence 8-9 still resulted in - negative 30d outcomes for insider_buying (-2.05% avg). High confidence scores - are overconfident across most strategies. -- Exception: minervini picks had 100% 1d win rate (4 data points), suggesting - score+confidence may be better calibrated for rule-based scanners vs. narrative-based. -- Confidence: medium (need more data to isolate score effect from strategy effect) +### 2026-04-12 — Cross-scanner calibration analysis +- All scanners show tight calibration: avg score/10 within 0.5 of avg confidence across all scanners. No systemic miscalibration. +- The current `min_score_threshold=55` in `discovery_config.py:52` allows borderline candidates (GME social_dd score 56, TSLA options_flow 60, FRT early_accumulation 60) into final rankings. +- These low-scoring picks carry confidence 5-6 and are explicitly speculative. Raising threshold to 65 would eliminate them without losing high-conviction picks. +- insider_buying has 136 recs — only 1 below score 60 (score 50-59 bucket had 1 entry). Raising to 65 would trim ~15% of insider picks (the 20 in 60-69 range). +- Confidence: medium ## Pending Hypotheses - [ ] Is confidence a better outcome predictor than final_score? -- [ ] Does score threshold (e.g. only surface candidates >70) improve hit rate? -- [ ] Does per-strategy score normalization help (e.g. social_dd score of 70 > insider score of 85)? +- [x] Does score threshold >65 improve hit rate? → Evidence supports it: low-score candidates are weak (social sentiment without data, speculative momentum). Implement threshold raise to 65. + +### 2026-04-12 — P&L outcome analysis (mature recs, 2nd iteration) +- news_catalyst: 0% 7d win rate, -8.79% avg 7d return (7 samples). Worst performing strategy by far. +- social_hype: 14.3% 7d win rate, -4.84% avg 7d, -10.45% avg 30d (21-22 samples). Consistent destroyer. +- social_dd: surprisingly best long-term: 55% 30d win rate, +0.94% avg 30d return — only scanner positive at 30d. +- minervini: best short-term signal but small sample (n=3 for 1d tracking). +- **Critical gap confirmed**: `format_stats_summary()` shows only top 3 best strategies. LLM never sees news_catalyst (0% 7d) or social_hype (14.3% 7d) as poor performers. +- Confidence: high diff --git a/docs/iterations/scanners/analyst_upgrades.md b/docs/iterations/scanners/analyst_upgrades.md index 767dbc70..efaa7987 100644 --- a/docs/iterations/scanners/analyst_upgrades.md +++ b/docs/iterations/scanners/analyst_upgrades.md @@ -7,8 +7,16 @@ target increase (>15%). Short squeeze potential (high short interest) combined w an upgrade is a historically strong setup. ## Evidence Log -_(populated by /iterate runs)_ + +### 2026-04-12 — P&L review + fast-loop +- 36 tracked recommendations (mature). Win rates: 38.2% 1d, 50.0% 7d, 30.4% 30d. Avg returns: +0.13% 1d, -0.75% 7d, -3.64% 30d. +- 7d win rate of 50% is close to coin-flip; 30d degrades sharply. +- Recent runs (Apr 6-12): 7 candidates — LRN, SEZL, NTWK, CSCO, NFLX, DLR, INTC. INTC Apr 12 (score=85) had a strong catalyst (Terafab + Apple rumor), which is a genuine material catalyst, fitting the "already priced in" concern. +- CSCO appeared in analyst_upgrade (Apr 8) AND options_flow (Apr 6, Apr 9) — cross-scanner confluence is a positive quality signal. +- Confidence calibration: Good (cal_diff ≤ 0.5 across all recent instances). +- Confidence: medium (36 samples, 7d win rate at breakeven) ## Pending Hypotheses - [ ] Does analyst tier (BB firm vs boutique) predict upgrade quality? - [ ] Does short interest >20% combined with an upgrade produce outsized moves? +- [ ] Does cross-scanner confluence (analyst_upgrade + options_flow on same ticker) predict higher 7d returns? diff --git a/docs/iterations/scanners/earnings_calendar.md b/docs/iterations/scanners/earnings_calendar.md index 1550247b..d59036ee 100644 --- a/docs/iterations/scanners/earnings_calendar.md +++ b/docs/iterations/scanners/earnings_calendar.md @@ -7,7 +7,15 @@ Standalone earnings calendar signal is too broad — nearly every stock has earn quarterly. ## Evidence Log -_(populated by /iterate runs)_ + +### 2026-04-12 — P&L review (earnings_play strategy, 65 tracked recs) +- Note: appears in statistics.json as "earnings_play" not "earnings_calendar". The scanner feeds this strategy. +- Win rates: 38.1% 1d, 37.7% 7d, 46.2% 30d. Avg returns: -0.33% 1d, -2.05% 7d, -2.8% 30d. +- The 30d win rate (46.2%) is better than 7d (37.7%) — unusual pattern suggesting the binary earnings event resolves negatively short-term but some recover. +- Recent runs: 4 candidates (APLD, SLP, FBK, FAST) all scored 60-75 — consistently lowest-scoring scanner in recent runs. APLD (score=75, high short interest 30.6%) is the strongest type of earnings_play setup. +- Avg scores in recent runs: 67 — below the 70 average for other scanners. The ranker is appropriately skeptical of this scanner. +- Confidence: high (65 samples with clear trend) ## Pending Hypotheses - [ ] Does requiring options confirmation alongside earnings improve signal quality? +- [ ] Does short interest >20% pre-earnings produce better outcomes than <10%? APLD (30.6% SI) scored highest in recent runs — worth tracking. diff --git a/docs/iterations/scanners/insider_buying.md b/docs/iterations/scanners/insider_buying.md index 62ddafa0..4443081b 100644 --- a/docs/iterations/scanners/insider_buying.md +++ b/docs/iterations/scanners/insider_buying.md @@ -6,23 +6,29 @@ Cluster detection (2+ insiders buying within 14 days) historically a high-convic setup. Transaction details (name, title, value) must be preserved from scraper output and included in candidate context — dropping them loses signal clarity. -Default `min_transaction_value` was $25K but P&L data (178 recs, -2.05% 30d avg) -indicates the low threshold allows sub-signal transactions through. Raised to $100K -to align with the registered insider_buying-min-txn-100k hypothesis. - ## Evidence Log -### 2026-04-11 — P&L review -- 178 recommendations over Feb–Apr 2026. Avg 30d return: -2.05%. 30d win rate: 29.4%. -- 1d win rate only 38.1%, suggesting price does not immediately react to filing disclosures. -- 7d win rate 46.3% — marginally better, but still below coin-flip at 30d. -- Sample files show most published recs had large transactions ($1M–$37M), but the - scanner's $25K floor likely admits many smaller, noisier transactions in the raw feed. -- Broader market context (tariff shock, sell-off Feb–Apr 2026) likely suppressed all - long signals, making it hard to isolate scanner quality from market conditions. -- Confidence: medium (market headwinds confound; need post-recovery data to isolate) +### 2026-04-12 — P&L review (2026-02-18 to 2026-04-07) +- insider_buying produced 136 recommendations — by far the highest volume scanner. +- Score distribution is healthy and concentrated: 53 picks in 80-89, 11 in 90-99, only 1 below 60. +- Confidence calibration is tight: avg score 78.6 (score/10 = 7.9) vs avg confidence 7.5 — well aligned. +- Cluster detection (2+ insiders → CRITICAL priority) is **already implemented** in code at `insider_buying.py:73`. The hypothesis was incorrect — this is live, not pending. +- High-conviction cluster examples surfaced: HMH (appeared in 2 separate runs Apr 8-9), FUL (Apr 9 and Apr 12), both with scores 71-82. +- Confidence: high + +### 2026-04-12 — Fast-loop (2026-04-08 to 2026-04-12) +- Insider_buying dominates final rankings: 3 of 6 ranked slots on Apr 9, 2 of 5 on Apr 10, contributing highest-ranked picks regularly. +- Context strings are specific and include insider name, title, dollar value — good signal clarity preserved. +- Confidence: high + +### 2026-04-12 — P&L update (180 tracked recs, mature data) +- Win rates are weaker than expected given high confidence scores: 38.1% 1d, 46.4% 7d, 29.7% 30d. +- Avg returns: -0.01% 1d, -0.4% 7d, -1.98% 30d — negative at every horizon. +- **Staleness pattern confirmed**: HMH appeared 4 consecutive days (Apr 6-9) with nearly identical scores (72, 85, 71, 82) — same insider filing, no new catalyst. FUL appeared Apr 9 and Apr 12 with identical scores (75). This is redundant signal, not confluence. +- High confidence (avg 7.1) combined with poor actual win rates = miscalibration — scanner assigns scores optimistically but real outcomes are below 50%. +- Confidence: high ## Pending Hypotheses -- [ ] Does cluster detection (2+ insiders in 14 days) outperform single-insider signals? -- [x] Is there a minimum transaction size below which signal quality degrades sharply? - → Raising threshold from $25K to $100K to test. Prior $25K baseline had -2.05% 30d avg. +- [x] Does cluster detection (2+ insiders in 14 days) outperform single-insider signals? → **Already implemented**: cluster detection assigns CRITICAL priority. Code verified at `insider_buying.py:73-74`. Cannot assess outcome vs single-insider yet (all statuses 'open'). +- [ ] Is there a minimum transaction size below which signal quality degrades sharply? (current min: $25K — candidates with $25K-$50K transactions show up at lower scores but still make final ranking) +- [ ] Does filtering out repeat appearances of the same ticker from the same scanner within 3 days improve precision? diff --git a/docs/iterations/scanners/minervini.md b/docs/iterations/scanners/minervini.md index bee28a97..26d31d84 100644 --- a/docs/iterations/scanners/minervini.md +++ b/docs/iterations/scanners/minervini.md @@ -6,20 +6,24 @@ uptrend, price above 50/150/200 SMA in the right order, 52-week high proximity, RS line at new highs. Historically one of the highest-conviction scanner setups. Works best in bull market conditions; underperforms in choppy/bear markets. -Early P&L evidence supports the high-conviction thesis: 100% 1d win rate and -+3.68% avg 1d return across 4 data points. No 7d/30d data available yet. -The market condition filter hypothesis remains untested. - ## Evidence Log -### 2026-04-11 — P&L review -- 4 recommendations. 1d win rate: 100%. Avg 1d return: +3.68%. -- No 7d or 30d data (positions still open or too recent at time of statistics cut). -- 4 data points is too small to draw conclusions but the signal is encouraging. -- Context: these 4 picks occurred during the broader Feb–Apr 2026 downturn, - suggesting the Stage 2 uptrend filter is effective at avoiding stocks in decline. -- Confidence: low (4 data points insufficient for statistical significance) +### 2026-04-12 — P&L review +- 7 tracked recommendations; 3/3 1-day wins measured, avg +3.68% 1d return. +- No 7d/30d data yet (too recent), but early 1d signal is strongest of all scanners. +- Recent week (Apr 6-12): 7 candidates produced — ALB (×2), AA (×2), AVGO (×2), BAC. Consistent quality signals. +- AA reappeared Apr 8 (score=68) then Apr 12 (score=92) — second appearance coincided with Morgan Stanley upgrade catalyst, showing scanner correctly elevated conviction when confluence added. +- Confidence calibration: Good (cal_diff ≤ 0.8 across all instances). +- Confidence: medium (small sample size, market was volatile Apr 6-12 due to tariff news) + +### 2026-04-12 — Fast-loop (2026-04-08 to 2026-04-12) +- minervini was top-ranked in 3 of 5 runs — highest hit-rate at #1 position of any scanner this week. +- AVGO ranked #1 on Apr 10 and Apr 11 (score 85, conf 8 both days) — persistent signal. +- Apr 2026 is risk-off (tariff volatility), yet Minervini setups are still leading. Contradicts bear-market underperformance assumption. +- Apr 12 AA thesis was highly specific: RS Rating 98, Morgan Stanley Overweight upgrade, earnings in 4 days, rising OBV. Good signal clarity. +- Confidence: high ## Pending Hypotheses -- [ ] Does adding a market condition filter (S&P 500 above 200 SMA) improve hit rate? -- [ ] Do RS Rating thresholds (>80 vs >90) meaningfully differentiate outcomes? +- [ ] Does adding a market condition filter (S&P 500 above 200 SMA) improve hit rate? Early evidence (Apr 2026 volatile market, still producing top picks) suggests filtering by market condition may hurt recall. +- [ ] Does a second appearance of the same ticker (persistence across days) predict higher returns than first-time appearances? +- [ ] Do earnings-nearby Minervini setups (within 5 days) underperform? Apr 12 AA has earnings in 4 days — flag for tracking. diff --git a/docs/iterations/scanners/options_flow.md b/docs/iterations/scanners/options_flow.md index 4f6720f8..527aed8a 100644 --- a/docs/iterations/scanners/options_flow.md +++ b/docs/iterations/scanners/options_flow.md @@ -3,29 +3,25 @@ ## Current Understanding Scans for unusual options volume relative to open interest using Tradier API. Call/put volume ratio below 0.1 is a reliable bullish signal when combined with -premium >$25K. The premium filter is configured but must be explicitly applied. +premium >$25K. The premium filter is applied at `options_flow.py:143-144`. Scanning only the nearest expiration misses institutional positioning in 30+ DTE contracts — scanning up to 3 expirations improves signal quality. -P&L data shows options_flow is underperforming at 30d (-2.86% avg, 29% win rate) -despite theoretically strong signal characteristics. Signal quality at 7d is -near-neutral (46.1% win rate), suggesting options flow predicts near-term moves -better than longer-term ones. - ## Evidence Log -### 2026-04-11 — P&L review -- 94 recommendations. 1d avg return: +0.03% (near flat). 7d avg: -0.91%. 30d avg: -2.86%. -- 7d win rate 46.1% is best of the poor strategies — nearly coin-flip, meaning the - direction signal has some validity but not enough edge to overcome transaction costs. -- 30d win rate drops to 29% — options flow signal appears to decay rapidly after ~1 week. -- Sample recommendations show P/C ratios of 0.02–0.48 (wide range); unclear if lower - P/C ratios (more bullish skew) predict better outcomes within this strategy. -- Hypothesis: the 7-day decay in win rate suggests options flow should be treated as - a short-horizon signal, not a basis for multi-week holds. +### 2026-04-12 — P&L review (2026-02-18 to 2026-04-07) +- options_flow produced 61 recommendations — second highest volume after insider_buying. +- Average score 74.7 (score/10 = 7.5), confidence 7.2 — well calibrated. +- The premium filter IS applied in code (`options_flow.py:143-144`): `(vol * price * 100) < self.min_premium` gates both calls and puts. "Premium filter configured but not explicitly applied" was incorrect — the hypothesis is resolved. +- CSCO appeared in options_flow on Apr 9 (score 85) and analyst_upgrade on Apr 8 (score 78) — cross-scanner confluence on same ticker. +- Confidence: high + +### 2026-04-12 — Fast-loop (2026-04-08 to 2026-04-12) +- options_flow appeared in 2 of 5 analyzed runs with CSCO and TSLA as the main picks. +- TSLA scored only 60 (conf 6) — borderline quality; appeared alongside GME social_dd (56) in same run (Apr 8), suggesting the LLM is rightly cautious about speculative social names. - Confidence: medium ## Pending Hypotheses +- [x] Premium filter: already applied in code at `options_flow.py:143-144, 159`. Hypothesis resolved. - [ ] Does scanning 3 expirations vs 1 meaningfully change hit rate? - [ ] Is moneyness (ITM vs OTM) a useful signal filter? -- [ ] Does P/C ratio below 0.1 (vs 0.1–0.5) predict significantly better 7d outcomes? diff --git a/tradingagents/dataflows/discovery/discovery_config.py b/tradingagents/dataflows/discovery/discovery_config.py index 63c7e6a7..19c3f78e 100644 --- a/tradingagents/dataflows/discovery/discovery_config.py +++ b/tradingagents/dataflows/discovery/discovery_config.py @@ -49,7 +49,7 @@ class RankerConfig: max_candidates_to_analyze: int = 200 analyze_all_candidates: bool = False final_recommendations: int = 15 - min_score_threshold: int = 55 + min_score_threshold: int = 65 return_target_pct: float = 5.0 holding_period_days: str = "1-7" truncate_ranking_context: bool = False