Multiple runs on the same day (scheduled discovery, hypothesis runner, manual
re-runs) were each clobbering the shared YYYY-MM-DD.json file. Now merges by
loading existing picks and upserting new ones by ticker — later run wins.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Requesting only candidate tickers produced a different cache key from the
nightly prefetch, triggering a fresh yfinance download that hit rate limits
(4/75 tickers returned). Now loads the full universe (same key as nightly
prefetch) and slices to candidate tickers — always a cache hit.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Change recent_mover_action default from "filter" to "deprioritize" so candidates
that moved >10% in the past 7 days reach the ranker (with lowered priority + context
annotation) rather than being silently dropped
- Exempt mean_reversion pipeline candidates (rsi_oversold) from the recent-mover action
entirely — stocks that pulled back hard are the signal, not a disqualifier
- Hoist SCANNER_REGISTRY import to module level (removes duplicate inline imports)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Change recent_mover_action default from "filter" to "deprioritize" so candidates
that moved >10% in the past 7 days reach the ranker (with lowered priority + context
annotation) rather than being silently dropped
- Exempt mean_reversion pipeline candidates (rsi_oversold) from the recent-mover action
entirely — stocks that pulled back hard are the signal, not a disqualifier
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The rsi_oversold scanner is a contrarian signal, not a momentum signal —
it fires on short-term panic pullbacks within uptrends and expects a 3-7 day
bounce. Keeping it in the momentum pipeline conflates two fundamentally
different signal types and could cause the ranker to misprice it.
Changes:
- Add mean_reversion pipeline (priority 6, deep_dive_budget 5) to default_config.py
- Move rsi_oversold scanner from pipeline="momentum" to pipeline="mean_reversion"
in both default_config.py and rsi_oversold.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- tradingagents/dataflows/universe.py: single source of truth for ticker
universe; all scanners now call load_universe(config) instead of
duplicating the 3-level fallback chain with hardcoded "data/tickers.txt"
- scripts/prefetch_ohlcv.py: nightly script using existing ohlcv_cache.py
incremental logic; first run downloads 1y history, subsequent runs append
only new trading days
- .github/workflows/prefetch.yml: runs at 01:00 UTC daily, before all other
workflows; commits updated parquet to repo
- Updated 6 scanners: minervini, high_52w_breakout, ml_signal, options_flow,
sector_rotation, technical_breakout — removed duplicate DEFAULT_TICKER_FILE
constants and _load_tickers_from_file() functions
- minervini, high_52w_breakout, technical_breakout: replace yf.download()
with download_ohlcv_cached() — reads from prefetched cache instead of
hitting yfinance at discovery time
- default_config.py: added discovery.ohlcv_cache_dir config key
- data/ohlcv_cache/: initial 1y backfill (588 tickers, 5.4MB parquet)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both scanners were implemented by research PRs but never added to default_config.
Without entries they cannot be disabled or tuned from config and are invisible
to the settings UI. Also updated /research-strategy to make config entry mandatory.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements ShortSqueezeScanner wrapping existing get_short_interest() in finviz_scraper.py.
Research finding: raw high SI predicts negative long-term returns (academic); edge is using
SI as a squeeze-risk flag when combined with earnings_calendar or options_flow catalysts.
Directly addresses earnings_calendar pending hypothesis (APLD 30.6% SI was strongest setup).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements ShortSqueezeScanner wrapping existing get_short_interest() in finviz_scraper.py.
Research finding: raw high SI predicts negative long-term returns (academic); edge is using
SI as a squeeze-risk flag when combined with earnings_calendar or options_flow catalysts.
Directly addresses earnings_calendar pending hypothesis (APLD 30.6% SI was strongest setup).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- active.json: updated days_elapsed from hypothesis runner
- hypotheses.py: black formatting applied by pre-commit hook
- .gitignore: local additions
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous implementation redirected sys.stderr to /dev/null using a
context manager. This is not thread-safe: 8 concurrent scanner threads each
mutate sys.stderr, and when one thread's context manager closes the devnull
file, another thread that captured devnull as its saved stderr attempts to
write to the closed fd and raises "I/O operation on closed file".
This corrupted sys.stderr state caused _fetch_batch_prices to fail and
all per-ticker get_stock_price fallback calls to return None, resulting in
every candidate being dropped with "no data available".
Fix by suppressing at the Python logging level instead of redirecting
sys.stderr. Logger.setLevel() is protected by internal locks and is safe
to call from concurrent threads.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs causing zero recommendations:
1. risk_metrics.py was untracked — importing it raised ModuleNotFoundError which
was caught by the outer try/except in filter.py, silently dropping all 32
candidates that reached the fundamental risk check stage.
2. Minervini scanner at max_tickers=200 took >5 min to download 200 tickers x 1y
of OHLCV data. ThreadPoolExecutor.cancel() cannot kill a running thread, so the
download kept running as a zombie thread for 20 more minutes after the pipeline
completed, holding the Python process alive until the 30-min workflow timeout
killed the entire job.
Reducing to 50 tickers brings the download to ~75s, well under the 300s global
scanner timeout.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The same-day mover filter used abs() to check intraday movement, which
filtered both gap-ups AND gap-downs. On volatile/crash days (e.g. 2026-04-07)
all stocks dropped >10% from open, causing every candidate to be filtered and
leaving zero recommendations.
The filter's purpose is to avoid chasing stocks that already ran up. A stock
down 20% intraday is not "stale" — it should be evaluated on its merits.
Changed threshold check from abs(pct) >= threshold to pct >= threshold so only
upside movers are filtered.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
yf.download(592 tickers, period=1y) takes 20+ minutes in CI, causing
the 30-minute job timeout to trigger. Add max_tickers=200 (configurable)
to limit the batch download to the first N tickers from the file. The
concurrent scanner pool already has a 5-min global timeout, but the hung
download thread monopolises network connections and starves the filter stage.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
minervini.py existed but was never committed. Without the file on the
remote, the __init__.py import added in the previous fix causes an
ImportError in CI.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add permissions: contents: write so git push works (was failing with 403)
- Add continue-on-error: true on discovery step so partial output still commits
- Change all commit/tracking/position steps to if: always() so they run regardless of discovery outcome
- Use commit-then-pull-rebase-then-push pattern to handle branch divergence
- Fix minervini scanner missing from scanners/__init__.py (enabled in config but never loaded)
- Fix .gitignore: results/* + !results/discovery/ so CI run logs can be committed
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Same issue as options_flow: early exit on candidate count discards strong
signals that happen to be later in iteration order.
insider_buying: Dict iteration order matched OpenInsider HTML scrape order,
not signal quality. Now scores by cluster buys + C-suite + dollar value,
then takes top N.
technical_breakout: Stopped at limit*2 in file order despite data already
being batch-downloaded (zero API cost to check all). Removed early exit,
scan full universe, sort by volume_multiple.
sector_rotation: Checked laggards in arbitrary dict order, spending API
calls on random tickers. Now sorts by most-negative 5d return first so
the strongest laggard candidates are checked before hitting the budget.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously the scanner stopped as soon as self.limit candidates were found
from as_completed() futures. Since futures complete in non-deterministic
network-latency order, this was equivalent to random sampling — fast-to-
respond tickers won regardless of how strong their options signal was.
Fix: collect all candidates from the full universe, then sort by options_score
(unusual strike count weighted 1.5x for calls to favor bullish flow) before
applying the limit. The top-N strongest signals are now always returned.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
tqdm writes to stderr immediately on __enter__, before any loop iteration.
In Streamlit's thread/subprocess context stderr can be a closed pipe, causing
'I/O operation on closed file' which _run_call catches and returns {} — so
the entire news enrichment step was silently skipped every run.
Replaced tqdm progress bars with logger.info() calls in:
- get_batch_stock_news_google() in openai.py
- get_batch_stock_news_openai() in openai.py
- Reddit DD parallel evaluation in reddit_api.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1. executor.shutdown(wait=True) still blocked after global timeout (critical)
The previous fix added timeout= to as_completed() but used `with
ThreadPoolExecutor() as executor`, whose __exit__ calls shutdown(wait=True).
This meant the process still hung waiting for stuck threads (ml_signal) even
after the TimeoutError was caught. Fixed by creating the executor explicitly
and calling shutdown(wait=False) in a finally block.
2. ml_signal hangs on every run — "Batch-downloading 592 tickers (1y)..." never
completes. Root cause: a single yfinance request for 592 tickers × 1 year of
daily OHLCV is a very large payload that regularly times out at the network
layer. Fixed by:
- Reducing default lookback from "1y" to "6mo" (halves download size)
- Splitting downloads into 150-ticker chunks so a slow chunk doesn't kill
the whole scan (partial results are still returned)
3. C (Citigroup) and other single-letter NYSE tickers rejected as invalid.
validate_ticker_format used ^[A-Z]{2,5}$ requiring at least 2 letters.
Real tickers like C, A, F, T, X, M are 1 letter. Fixed to ^[A-Z]{1,5}$.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Youssef Aitousarrah
Aitous
github-actions[bot]