feat: implement data-driven external data integration (issue #22)
Add objective external data sources to enhance trading decisions beyond market prices and user input. ## New Modules ### src/data/news_api.py - News sentiment analysis with Alpha Vantage and NewsAPI support - Sentiment scoring (-1.0 to +1.0) per article and aggregated - 5-minute caching to minimize API quota usage - Graceful degradation when APIs unavailable ### src/data/economic_calendar.py - Track major economic events (FOMC, GDP, CPI) - Earnings calendar per stock - Event proximity checking for high-volatility periods - Hardcoded major events for 2026 (no API required) ### src/data/market_data.py - Market sentiment indicators (Fear & Greed equivalent) - Market breadth (advance/decline ratios) - Sector performance tracking - Fear/Greed score calculation ## Integration Enhanced GeminiClient to seamlessly integrate external data: - Optional news_api, economic_calendar, and market_data parameters - Async build_prompt() includes external context when available - Backward-compatible build_prompt_sync() for existing code - Graceful fallback when external data unavailable External data automatically added to AI prompts: - News sentiment with top articles - Upcoming high-impact economic events - Market sentiment and breadth indicators ## Configuration Added optional settings to config.py: - NEWS_API_KEY: API key for news provider - NEWS_API_PROVIDER: "alphavantage" or "newsapi" - MARKET_DATA_API_KEY: API key for market data ## Testing Comprehensive test suite with 38 tests: - NewsAPI caching, sentiment parsing, API integration - EconomicCalendar event filtering, earnings lookup - MarketData sentiment and breadth calculations - GeminiClient integration with external data sources - All tests use mocks (no real API keys required) - 81% coverage for src/data module (exceeds 80% requirement) ## Circular Import Fix Fixed circular dependency between gemini_client.py and cache.py: - Use TYPE_CHECKING for imports in cache.py - String annotations for TradeDecision type hints All 195 existing tests pass. No breaking changes to existing functionality. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
205
src/data/README.md
Normal file
205
src/data/README.md
Normal file
@@ -0,0 +1,205 @@
|
||||
# External Data Integration
|
||||
|
||||
This module provides objective external data sources to enhance trading decisions beyond just market prices and user input.
|
||||
|
||||
## Modules
|
||||
|
||||
### `news_api.py` - News Sentiment Analysis
|
||||
|
||||
Fetches real-time news for stocks with sentiment scoring.
|
||||
|
||||
**Features:**
|
||||
- Alpha Vantage and NewsAPI.org support
|
||||
- Sentiment scoring (-1.0 to +1.0)
|
||||
- 5-minute caching to minimize API quota usage
|
||||
- Graceful fallback when API unavailable
|
||||
|
||||
**Usage:**
|
||||
```python
|
||||
from src.data.news_api import NewsAPI
|
||||
|
||||
# Initialize with API key
|
||||
news_api = NewsAPI(api_key="your_key", provider="alphavantage")
|
||||
|
||||
# Fetch news sentiment
|
||||
sentiment = await news_api.get_news_sentiment("AAPL")
|
||||
if sentiment:
|
||||
print(f"Average sentiment: {sentiment.avg_sentiment}")
|
||||
for article in sentiment.articles[:3]:
|
||||
print(f"{article.title} ({article.sentiment_score})")
|
||||
```
|
||||
|
||||
### `economic_calendar.py` - Major Economic Events
|
||||
|
||||
Tracks FOMC meetings, GDP releases, CPI, earnings calendars, and other market-moving events.
|
||||
|
||||
**Features:**
|
||||
- High-impact event tracking (FOMC, GDP, CPI)
|
||||
- Earnings calendar per stock
|
||||
- Event proximity checking
|
||||
- Hardcoded major events for 2026 (no API required)
|
||||
|
||||
**Usage:**
|
||||
```python
|
||||
from src.data.economic_calendar import EconomicCalendar
|
||||
|
||||
calendar = EconomicCalendar()
|
||||
calendar.load_hardcoded_events()
|
||||
|
||||
# Get upcoming high-impact events
|
||||
upcoming = calendar.get_upcoming_events(days_ahead=7, min_impact="HIGH")
|
||||
print(f"High-impact events: {upcoming.high_impact_count}")
|
||||
|
||||
# Check if near earnings
|
||||
earnings_date = calendar.get_earnings_date("AAPL")
|
||||
if earnings_date:
|
||||
print(f"Next earnings: {earnings_date}")
|
||||
|
||||
# Check for high volatility period
|
||||
if calendar.is_high_volatility_period(hours_ahead=24):
|
||||
print("High-impact event imminent!")
|
||||
```
|
||||
|
||||
### `market_data.py` - Market Indicators
|
||||
|
||||
Provides market breadth, sector performance, and sentiment indicators.
|
||||
|
||||
**Features:**
|
||||
- Market sentiment levels (Fear & Greed equivalent)
|
||||
- Market breadth (advancing/declining stocks)
|
||||
- Sector performance tracking
|
||||
- Fear/Greed score calculation
|
||||
|
||||
**Usage:**
|
||||
```python
|
||||
from src.data.market_data import MarketData
|
||||
|
||||
market_data = MarketData(api_key="your_key")
|
||||
|
||||
# Get market sentiment
|
||||
sentiment = market_data.get_market_sentiment()
|
||||
print(f"Market sentiment: {sentiment.name}")
|
||||
|
||||
# Get full indicators
|
||||
indicators = market_data.get_market_indicators("US")
|
||||
print(f"Sentiment: {indicators.sentiment.name}")
|
||||
print(f"A/D Ratio: {indicators.breadth.advance_decline_ratio}")
|
||||
```
|
||||
|
||||
## Integration with GeminiClient
|
||||
|
||||
The external data sources are seamlessly integrated into the AI decision engine:
|
||||
|
||||
```python
|
||||
from src.brain.gemini_client import GeminiClient
|
||||
from src.data.news_api import NewsAPI
|
||||
from src.data.economic_calendar import EconomicCalendar
|
||||
from src.data.market_data import MarketData
|
||||
from src.config import Settings
|
||||
|
||||
settings = Settings()
|
||||
|
||||
# Initialize data sources
|
||||
news_api = NewsAPI(api_key=settings.NEWS_API_KEY, provider=settings.NEWS_API_PROVIDER)
|
||||
calendar = EconomicCalendar()
|
||||
calendar.load_hardcoded_events()
|
||||
market_data = MarketData(api_key=settings.MARKET_DATA_API_KEY)
|
||||
|
||||
# Create enhanced client
|
||||
client = GeminiClient(
|
||||
settings,
|
||||
news_api=news_api,
|
||||
economic_calendar=calendar,
|
||||
market_data=market_data
|
||||
)
|
||||
|
||||
# Make decision with external context
|
||||
market_data_dict = {
|
||||
"stock_code": "AAPL",
|
||||
"current_price": 180.0,
|
||||
"market_name": "US stock market"
|
||||
}
|
||||
|
||||
decision = await client.decide(market_data_dict)
|
||||
```
|
||||
|
||||
The external data is automatically included in the prompt sent to Gemini:
|
||||
|
||||
```
|
||||
Market: US stock market
|
||||
Stock Code: AAPL
|
||||
Current Price: 180.0
|
||||
|
||||
EXTERNAL DATA:
|
||||
News Sentiment: 0.85 (from 10 articles)
|
||||
1. [Reuters] Apple hits record high (sentiment: 0.92)
|
||||
2. [Bloomberg] Strong iPhone sales (sentiment: 0.78)
|
||||
3. [CNBC] Tech sector rallying (sentiment: 0.85)
|
||||
|
||||
Upcoming High-Impact Events: 2 in next 7 days
|
||||
Next: FOMC Meeting (FOMC) on 2026-03-18
|
||||
Earnings: AAPL on 2026-02-10
|
||||
|
||||
Market Sentiment: GREED
|
||||
Advance/Decline Ratio: 2.35
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Add these to your `.env` file:
|
||||
|
||||
```bash
|
||||
# External Data APIs (optional)
|
||||
NEWS_API_KEY=your_alpha_vantage_key
|
||||
NEWS_API_PROVIDER=alphavantage # or "newsapi"
|
||||
MARKET_DATA_API_KEY=your_market_data_key
|
||||
```
|
||||
|
||||
## API Recommendations
|
||||
|
||||
### Alpha Vantage (News)
|
||||
- **Free tier:** 5 calls/min, 500 calls/day
|
||||
- **Pros:** Provides sentiment scores, no credit card required
|
||||
- **URL:** https://www.alphavantage.co/
|
||||
|
||||
### NewsAPI.org
|
||||
- **Free tier:** 100 requests/day
|
||||
- **Pros:** Large news coverage, easy to use
|
||||
- **Cons:** No sentiment scores (we use keyword heuristics)
|
||||
- **URL:** https://newsapi.org/
|
||||
|
||||
## Caching Strategy
|
||||
|
||||
To minimize API quota usage:
|
||||
|
||||
1. **News:** 5-minute TTL cache per stock
|
||||
2. **Economic Calendar:** Loaded once at startup (hardcoded events)
|
||||
3. **Market Data:** Fetched per decision (lightweight)
|
||||
|
||||
## Graceful Degradation
|
||||
|
||||
The system works gracefully without external data:
|
||||
|
||||
- If no API keys provided → decisions work with just market prices
|
||||
- If API fails → decision continues without external context
|
||||
- If cache expired → attempts refetch, falls back to no data
|
||||
- Errors are logged but never block trading decisions
|
||||
|
||||
## Testing
|
||||
|
||||
All modules have comprehensive test coverage (81%+):
|
||||
|
||||
```bash
|
||||
pytest tests/test_data_integration.py -v --cov=src/data
|
||||
```
|
||||
|
||||
Tests use mocks to avoid requiring real API keys.
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- Twitter/X sentiment analysis
|
||||
- Reddit WallStreetBets sentiment
|
||||
- Options flow data
|
||||
- Insider trading activity
|
||||
- Analyst upgrades/downgrades
|
||||
- Real-time economic data APIs
|
||||
Reference in New Issue
Block a user