CipherTrade

Autonomous Cryptocurrency Trading using Soft Actor-Critic Reinforcement Learning

CipherTrade is an end-to-end deep RL trading system that trains a SAC agent with a CNN encoder to trade BTC/USD, with a live Streamlit dashboard for monitoring and a full MLflow experiment tracking pipeline.

---

Features

• SAC Agent : Soft Actor-Critic with automatic entropy tuning, twin critics, and gradient clipping
• CNN Encoder : Image-based market state representation (128×128 candlestick renders) processed by a deep convolutional encoder
• Custom Gym Environment : CipherTradeEnv with configurable commission, observation horizon, and opening balance
• Live Streamlit Dashboard : Real-time candlestick chart, portfolio equity curve, trade log, and episode metrics
• MLflow Tracking : All hyperparameters, per-episode metrics, and model checkpoints logged automatically

---

Project Structure

CipherTrade/
├── app.py                  # Streamlit dashboard
├── sac_agent.py            # SAC agent + CNN actor/critic + training loop
├── crypto_trading_env.py   # Custom Gymnasium trading environment
├── models/                 # Saved actor weights (.h5)
└── mlflow_store/           # MLflow DB + artifacts

---

Quickstart

1. Install dependencies

pip install -r requirements.txt

2. Run the dashboard (demo mode)

streamlit run app.py

Use the Random Agent mode to see the environment in action without training.

3. Train the SAC agent

python sac_agent.py

Training runs for 500 episodes by default, saving checkpoints every 50 episodes and the best actor weights to models/best_actor.weights.h5.

4. Run with a trained agent

Launch the dashboard, select Trained Agent, and point it to your saved .weights.h5 file.

---

Architecture

Observation (128×128×3 candlestick image)
        │
   CNN Encoder (3 Conv layers → Dense 256)
        │
   ┌────┴────┐
 Actor     Critic1 + Critic2
(μ, σ)       (Q-values)
   │              │
  Action     Bellman target
        │
  Environment Step

The actor outputs a mean and log-std; actions are sampled via the reparameterisation trick and squashed through tanh. The twin-critic setup prevents Q-value overestimation. Alpha (temperature) is tuned automatically to match a target entropy of -action_dim.

---

Configuration

Key hyperparameters are set in SACAgent.__init__() and logged to MLflow:

Parameter	Default	Description
lr_actor	3e-4	Actor learning rate
lr_critic	3e-4	Critic learning rate
gamma	0.99	Discount factor
tau	0.005	Soft target update rate
batch_size	64	Replay batch size
memory_cap	50,000	Replay buffer capacity
warmup_steps	1,000	Random exploration before learning
grad_clip	1.0	Global gradient norm clip

Dashboard parameters (opening balance, commission, horizon, max steps) are configurable via the sidebar at runtime.

---

MLflow Tracking

Metrics logged per episode:

• episode_reward
• portfolio_value
• pnl_pct
• alpha (entropy temperature)
• critic1_loss, critic2_loss, actor_loss
• steps

To view the MLflow UI locally:

mlflow ui --port 5000 --backend-store-uri sqlite:///mlflow_store/mlflow.db --default-artifact-root ./mlflow_store/artifacts

---

Environment Details

CipherTradeEnv is a custom Gymnasium environment that:

• Pulls BTC/USD OHLCV data via yfinance, with Yahoo Finance direct CSV and Binance public API as sequential fallbacks; results are cached locally for 24 hours
• Renders candlestick images as the observation
• Supports configurable commission rates and observation horizons
• Tracks cash, coins held, portfolio value, and trade history
• Returns an info dict with portfolio_value, cash, and coins_held at each step

---