feat: implement L1-L7 context tree for multi-layered memory management
Some checks failed
CI / test (pull_request) Has been cancelled
Some checks failed
CI / test (pull_request) Has been cancelled
Implements Pillar 2 (Multi-layered Context Management) with a 7-tier hierarchical memory system from real-time market data to generational trading wisdom. ## New Modules - `src/context/layer.py`: ContextLayer enum and metadata config - `src/context/store.py`: ContextStore for CRUD operations - `src/context/aggregator.py`: Bottom-up aggregation (L7→L6→...→L1) ## Database Changes - Added `contexts` table for hierarchical data storage - Added `context_metadata` table for layer configuration - Indexed by layer, timeframe, and updated_at for fast queries ## Context Layers - L1 (Legacy): Cumulative wisdom (kept forever) - L2 (Annual): Yearly metrics (10 years retention) - L3 (Quarterly): Strategy pivots (3 years) - L4 (Monthly): Portfolio rebalancing (2 years) - L5 (Weekly): Stock selection (1 year) - L6 (Daily): Trade logs (90 days) - L7 (Real-time): Live market data (7 days) ## Tests - 18 new tests in `tests/test_context.py` - 100% coverage on context modules - All 72 tests passing (54 existing + 18 new) ## Documentation - Added `docs/context-tree.md` with comprehensive guide - Updated `CLAUDE.md` architecture section - Includes usage examples and best practices Closes #15 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
agentson
338
docs/context-tree.md
Normal file
338
docs/context-tree.md
Normal file
@@ -0,0 +1,338 @@
|
||||
# Context Tree: Multi-Layered Memory Management
|
||||
|
||||
The context tree implements **Pillar 2** of The Ouroboros: hierarchical memory management across 7 time horizons, from real-time market data to generational trading wisdom.
|
||||
|
||||
## Overview
|
||||
|
||||
Instead of a flat memory structure, The Ouroboros maintains a **7-tier context tree** where each layer represents a different time horizon and level of abstraction:
|
||||
|
||||
```
|
||||
L1 (Legacy) ← Cumulative wisdom across generations
|
||||
↑
|
||||
L2 (Annual) ← Yearly performance metrics
|
||||
↑
|
||||
L3 (Quarterly) ← Quarterly strategy adjustments
|
||||
↑
|
||||
L4 (Monthly) ← Monthly portfolio rebalancing
|
||||
↑
|
||||
L5 (Weekly) ← Weekly stock selection
|
||||
↑
|
||||
L6 (Daily) ← Daily trade logs
|
||||
↑
|
||||
L7 (Real-time) ← Live market data
|
||||
```
|
||||
|
||||
Data flows **bottom-up**: real-time trades aggregate into daily summaries, which roll up to weekly, then monthly, quarterly, annual, and finally into permanent legacy knowledge.
|
||||
|
||||
## The 7 Layers
|
||||
|
||||
### L7: Real-time
|
||||
**Retention**: 7 days
|
||||
**Timeframe format**: `YYYY-MM-DD` (same-day)
|
||||
**Content**: Current positions, live quotes, orderbook snapshots, tick-by-tick volatility
|
||||
|
||||
**Use cases**:
|
||||
- Immediate execution decisions
|
||||
- Stop-loss triggers
|
||||
- Real-time P&L tracking
|
||||
|
||||
**Example keys**:
|
||||
- `current_position_{stock_code}`: Current holdings
|
||||
- `live_price_{stock_code}`: Latest quote
|
||||
- `volatility_5m_{stock_code}`: 5-minute rolling volatility
|
||||
|
||||
### L6: Daily
|
||||
**Retention**: 90 days
|
||||
**Timeframe format**: `YYYY-MM-DD`
|
||||
**Content**: Daily trade logs, end-of-day P&L, market summaries, decision accuracy
|
||||
|
||||
**Use cases**:
|
||||
- Daily performance review
|
||||
- Identify patterns in recent trading
|
||||
- Backtest strategy adjustments
|
||||
|
||||
**Example keys**:
|
||||
- `total_pnl`: Daily profit/loss
|
||||
- `trade_count`: Number of trades
|
||||
- `win_rate`: Percentage of profitable trades
|
||||
- `avg_confidence`: Average Gemini confidence
|
||||
|
||||
### L5: Weekly
|
||||
**Retention**: 1 year
|
||||
**Timeframe format**: `YYYY-Www` (ISO week, e.g., `2026-W06`)
|
||||
**Content**: Weekly stock selection, sector rotation, volatility regime classification
|
||||
|
||||
**Use cases**:
|
||||
- Weekly strategy adjustment
|
||||
- Sector momentum tracking
|
||||
- Identify hot/cold markets
|
||||
|
||||
**Example keys**:
|
||||
- `weekly_pnl`: Week's total P&L
|
||||
- `top_performers`: Best-performing stocks
|
||||
- `sector_focus`: Dominant sectors
|
||||
- `avg_confidence`: Weekly average confidence
|
||||
|
||||
### L4: Monthly
|
||||
**Retention**: 2 years
|
||||
**Timeframe format**: `YYYY-MM`
|
||||
**Content**: Monthly portfolio rebalancing, risk exposure analysis, drawdown recovery
|
||||
|
||||
**Use cases**:
|
||||
- Monthly performance reporting
|
||||
- Risk exposure adjustment
|
||||
- Correlation analysis
|
||||
|
||||
**Example keys**:
|
||||
- `monthly_pnl`: Month's total P&L
|
||||
- `sharpe_ratio`: Risk-adjusted return
|
||||
- `max_drawdown`: Largest peak-to-trough decline
|
||||
- `rebalancing_notes`: Manual insights
|
||||
|
||||
### L3: Quarterly
|
||||
**Retention**: 3 years
|
||||
**Timeframe format**: `YYYY-Qn` (e.g., `2026-Q1`)
|
||||
**Content**: Quarterly strategy pivots, market phase detection (bull/bear/sideways), macro regime changes
|
||||
|
||||
**Use cases**:
|
||||
- Strategic pivots (e.g., growth → value)
|
||||
- Macro regime classification
|
||||
- Long-term pattern recognition
|
||||
|
||||
**Example keys**:
|
||||
- `quarterly_pnl`: Quarter's total P&L
|
||||
- `market_phase`: Bull/Bear/Sideways
|
||||
- `strategy_adjustments`: Major changes made
|
||||
- `lessons_learned`: Key insights
|
||||
|
||||
### L2: Annual
|
||||
**Retention**: 10 years
|
||||
**Timeframe format**: `YYYY`
|
||||
**Content**: Yearly returns, Sharpe ratio, max drawdown, win rate, strategy effectiveness
|
||||
|
||||
**Use cases**:
|
||||
- Annual performance review
|
||||
- Multi-year trend analysis
|
||||
- Strategy benchmarking
|
||||
|
||||
**Example keys**:
|
||||
- `annual_pnl`: Year's total P&L
|
||||
- `sharpe_ratio`: Annual risk-adjusted return
|
||||
- `win_rate`: Yearly win percentage
|
||||
- `best_strategy`: Most successful strategy
|
||||
- `worst_mistake`: Biggest lesson learned
|
||||
|
||||
### L1: Legacy
|
||||
**Retention**: Forever
|
||||
**Timeframe format**: `LEGACY` (single timeframe)
|
||||
**Content**: Cumulative trading history, core principles, generational wisdom
|
||||
|
||||
**Use cases**:
|
||||
- Long-term philosophy
|
||||
- Foundational rules
|
||||
- Lessons that transcend market cycles
|
||||
|
||||
**Example keys**:
|
||||
- `total_pnl`: All-time profit/loss
|
||||
- `years_traded`: Trading longevity
|
||||
- `avg_annual_pnl`: Long-term average return
|
||||
- `core_principles`: Immutable trading rules
|
||||
- `greatest_trades`: Hall of fame
|
||||
- `never_again`: Permanent warnings
|
||||
|
||||
## Usage
|
||||
|
||||
### Setting Context
|
||||
|
||||
```python
|
||||
from src.context import ContextLayer, ContextStore
|
||||
from src.db import init_db
|
||||
|
||||
conn = init_db("data/ouroboros.db")
|
||||
store = ContextStore(conn)
|
||||
|
||||
# Store daily P&L
|
||||
store.set_context(
|
||||
layer=ContextLayer.L6_DAILY,
|
||||
timeframe="2026-02-04",
|
||||
key="total_pnl",
|
||||
value=1234.56
|
||||
)
|
||||
|
||||
# Store weekly insight
|
||||
store.set_context(
|
||||
layer=ContextLayer.L5_WEEKLY,
|
||||
timeframe="2026-W06",
|
||||
key="top_performers",
|
||||
value=["005930", "000660", "035720"] # JSON-serializable
|
||||
)
|
||||
|
||||
# Store legacy wisdom
|
||||
store.set_context(
|
||||
layer=ContextLayer.L1_LEGACY,
|
||||
timeframe="LEGACY",
|
||||
key="core_principles",
|
||||
value=[
|
||||
"Cut losses fast",
|
||||
"Let winners run",
|
||||
"Never average down on losing positions"
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
### Retrieving Context
|
||||
|
||||
```python
|
||||
# Get a specific value
|
||||
pnl = store.get_context(ContextLayer.L6_DAILY, "2026-02-04", "total_pnl")
|
||||
# Returns: 1234.56
|
||||
|
||||
# Get all keys for a timeframe
|
||||
daily_summary = store.get_all_contexts(ContextLayer.L6_DAILY, "2026-02-04")
|
||||
# Returns: {"total_pnl": 1234.56, "trade_count": 10, "win_rate": 60.0, ...}
|
||||
|
||||
# Get all data for a layer (any timeframe)
|
||||
all_daily = store.get_all_contexts(ContextLayer.L6_DAILY)
|
||||
# Returns: {"total_pnl": 1234.56, "trade_count": 10, ...} (latest timeframes first)
|
||||
|
||||
# Get the latest timeframe
|
||||
latest = store.get_latest_timeframe(ContextLayer.L6_DAILY)
|
||||
# Returns: "2026-02-04"
|
||||
```
|
||||
|
||||
### Automatic Aggregation
|
||||
|
||||
The `ContextAggregator` rolls up data from lower to higher layers:
|
||||
|
||||
```python
|
||||
from src.context.aggregator import ContextAggregator
|
||||
|
||||
aggregator = ContextAggregator(conn)
|
||||
|
||||
# Aggregate daily metrics from trades
|
||||
aggregator.aggregate_daily_from_trades("2026-02-04")
|
||||
|
||||
# Roll up weekly from daily
|
||||
aggregator.aggregate_weekly_from_daily("2026-W06")
|
||||
|
||||
# Roll up all layers at once (bottom-up)
|
||||
aggregator.run_all_aggregations()
|
||||
```
|
||||
|
||||
**Aggregation schedule** (recommended):
|
||||
- **L7 → L6**: Every midnight (daily rollup)
|
||||
- **L6 → L5**: Every Sunday (weekly rollup)
|
||||
- **L5 → L4**: First day of each month (monthly rollup)
|
||||
- **L4 → L3**: First day of quarter (quarterly rollup)
|
||||
- **L3 → L2**: January 1st (annual rollup)
|
||||
- **L2 → L1**: On demand (major milestones)
|
||||
|
||||
### Context Cleanup
|
||||
|
||||
Expired contexts are automatically deleted based on retention policies:
|
||||
|
||||
```python
|
||||
# Manual cleanup
|
||||
deleted = store.cleanup_expired_contexts()
|
||||
# Returns: {ContextLayer.L7_REALTIME: 42, ContextLayer.L6_DAILY: 15, ...}
|
||||
```
|
||||
|
||||
**Retention policies** (defined in `src/context/layer.py`):
|
||||
- L1: Forever
|
||||
- L2: 10 years
|
||||
- L3: 3 years
|
||||
- L4: 2 years
|
||||
- L5: 1 year
|
||||
- L6: 90 days
|
||||
- L7: 7 days
|
||||
|
||||
## Integration with Gemini Brain
|
||||
|
||||
The context tree provides hierarchical memory for decision-making:
|
||||
|
||||
```python
|
||||
from src.brain.gemini_client import GeminiClient
|
||||
|
||||
# Build prompt with multi-layer context
|
||||
def build_enhanced_prompt(stock_code: str, store: ContextStore) -> str:
|
||||
# L7: Real-time data
|
||||
current_price = store.get_context(ContextLayer.L7_REALTIME, "2026-02-04", f"live_price_{stock_code}")
|
||||
|
||||
# L6: Recent daily performance
|
||||
yesterday_pnl = store.get_context(ContextLayer.L6_DAILY, "2026-02-03", "total_pnl")
|
||||
|
||||
# L5: Weekly trend
|
||||
weekly_data = store.get_all_contexts(ContextLayer.L5_WEEKLY, "2026-W06")
|
||||
|
||||
# L1: Core principles
|
||||
principles = store.get_context(ContextLayer.L1_LEGACY, "LEGACY", "core_principles")
|
||||
|
||||
return f"""
|
||||
Analyze {stock_code} for trading decision.
|
||||
|
||||
Current price: {current_price}
|
||||
Yesterday's P&L: {yesterday_pnl}
|
||||
This week: {weekly_data}
|
||||
|
||||
Core principles:
|
||||
{chr(10).join(f'- {p}' for p in principles)}
|
||||
|
||||
Decision (BUY/SELL/HOLD):
|
||||
"""
|
||||
```
|
||||
|
||||
## Database Schema
|
||||
|
||||
```sql
|
||||
-- Context storage
|
||||
CREATE TABLE contexts (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
layer TEXT NOT NULL, -- L1_LEGACY, L2_ANNUAL, ..., L7_REALTIME
|
||||
timeframe TEXT NOT NULL, -- "LEGACY", "2026", "2026-Q1", "2026-02", "2026-W06", "2026-02-04"
|
||||
key TEXT NOT NULL, -- "total_pnl", "win_rate", "core_principles", etc.
|
||||
value TEXT NOT NULL, -- JSON-serialized value
|
||||
created_at TEXT NOT NULL, -- ISO 8601 timestamp
|
||||
updated_at TEXT NOT NULL, -- ISO 8601 timestamp
|
||||
UNIQUE(layer, timeframe, key)
|
||||
);
|
||||
|
||||
-- Layer metadata
|
||||
CREATE TABLE context_metadata (
|
||||
layer TEXT PRIMARY KEY,
|
||||
description TEXT NOT NULL,
|
||||
retention_days INTEGER, -- NULL = keep forever
|
||||
aggregation_source TEXT -- Parent layer for rollup
|
||||
);
|
||||
|
||||
-- Indices for fast queries
|
||||
CREATE INDEX idx_contexts_layer ON contexts(layer);
|
||||
CREATE INDEX idx_contexts_timeframe ON contexts(timeframe);
|
||||
CREATE INDEX idx_contexts_updated ON contexts(updated_at);
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Write to leaf layers only** — Never manually write to L1-L5; let aggregation populate them
|
||||
2. **Aggregate regularly** — Schedule aggregation jobs to keep higher layers fresh
|
||||
3. **Query specific timeframes** — Use `get_context(layer, timeframe, key)` for precise retrieval
|
||||
4. **Clean up periodically** — Run `cleanup_expired_contexts()` weekly to free space
|
||||
5. **Preserve L1 forever** — Legacy wisdom should never expire
|
||||
6. **Use JSON-serializable values** — Store dicts, lists, strings, numbers (not custom objects)
|
||||
|
||||
## Testing
|
||||
|
||||
See `tests/test_context.py` for comprehensive test coverage (18 tests, 100% coverage on context modules).
|
||||
|
||||
```bash
|
||||
pytest tests/test_context.py -v
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- **Implementation**: `src/context/`
|
||||
- `layer.py`: Layer definitions and metadata
|
||||
- `store.py`: CRUD operations
|
||||
- `aggregator.py`: Bottom-up aggregation logic
|
||||
- **Database**: `src/db.py` (table initialization)
|
||||
- **Tests**: `tests/test_context.py`
|
||||
- **Related**: Pillar 2 (Multi-layered Context Management)
|
||||
Reference in New Issue
Block a user