Skip to content

Commit c1fa4e0

Browse files
JOYclaude
JOY
and
claude
committed
docs: add DOS.AI system architecture document
Covers API gateway two-layer billing model (application-layer request-based vs gateway token-based), self-hosted GPU fleet, LLM fallback chain with cost tracking, and key type reference. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 55000eb commit c1fa4e0

1 file changed

Lines changed: 185 additions & 0 deletions

File tree

‎DOS.AI-Architecture.md‎

Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,185 @@
1+
# DOS.AI — System Architecture
2+
3+
**Updated:** 2026-03-14
4+
**Status:** API Gateway Phase 0 complete. InferenceSense MVP draft.
5+
6+
---
7+
8+
## What is DOS.AI?
9+
10+
DOS.AI is the AI infrastructure layer of the DOS ecosystem. It provides:
11+
12+
1. **Self-hosted LLM inference** — GPU fleet running open-source models (Qwen3.5)
13+
2. **API Gateway** — unified endpoint at `api.dos.ai` for all LLM consumers
14+
3. **Enterprise API** — key management, billing, usage tracking for external developers
15+
16+
## Architecture Overview
17+
18+
```
19+
┌──────────────────────────────────────────────────────────────────┐
20+
│ CONSUMERS │
21+
│ │
22+
│ ┌─────────────┐ ┌─────────────┐ ┌──────────────────────────┐ │
23+
│ │ DOSafe │ │ DOS.AI App │ │ External Developers │ │
24+
│ │ (internal) │ │ (internal) │ │ (dos_sk_xxx keys) │ │
25+
│ └──────┬───────┘ └──────┬──────┘ └──────────┬───────────────┘ │
26+
│ │ │ │ │
27+
│ INTERNAL_API_KEY INTERNAL_API_KEY dos_sk_xxx │
28+
│ (bypass billing) (bypass billing) (token-based billing) │
29+
│ │ │ │ │
30+
│ └──────────────────┼─────────────────────┘ │
31+
│ ▼ │
32+
│ ┌────────────────────────────────────────────────────────────┐ │
33+
│ │ api.dos.ai — Cloudflare Worker │ │
34+
│ │ │ │
35+
│ │ Auth: INTERNAL_API_KEY → skip billing │ │
36+
│ │ dos_sk_xxx → D1 key lookup + token billing │ │
37+
│ │ Route: self-hosted first → paid fallback │ │
38+
│ │ Meter: log tokens, latency, provider per request │ │
39+
│ │ Bill: deductBalance for dos_sk_xxx keys only │ │
40+
│ └──────────────────────┬─────────────────────────────────────┘ │
41+
│ │ │
42+
│ ┌────────────────┼────────────────────┐ │
43+
│ ▼ ▼ ▼ │
44+
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐ │
45+
│ │ Self-hosted │ │ Alibaba │ │ Google / OpenAI │ │
46+
│ │ vLLM fleet │ │ Cloud │ │ (future providers) │ │
47+
│ │ (priority 1) │ │ (priority 2)│ │ (priority 3+) │ │
48+
│ │ cost: $0 │ │ $0.10/$0.40 │ │ varies │ │
49+
│ └──────────────┘ └──────────────┘ └──────────────────────┘ │
50+
└──────────────────────────────────────────────────────────────────┘
51+
```
52+
53+
---
54+
55+
## Two-Layer Billing Model
56+
57+
The gateway serves two consumer types with different billing strategies:
58+
59+
### Layer 1 — Application Layer (request-based)
60+
61+
Internal DOS products (DOSafe, DOS.AI app) handle their own user-facing billing **before** calling the gateway.
62+
63+
| Product | Billing method | Implementation |
64+
|---------|---------------|----------------|
65+
| DOSafe Telegram bot | `consume_quota()` RPC per request | `dosafe.bot_quota` + `dosafe.client_quota` |
66+
| DOSafe web app | `dosafe_usage` per request | Supabase RPC |
67+
| DOSafe Chrome extension | Anonymous IP-based quota | Supabase RPC |
68+
| DOS.AI app features | Per-feature billing (TBD) | — |
69+
70+
These products authenticate with `INTERNAL_API_KEY` → gateway skips billing, rate limiting, and usage tracking. The gateway is pure infrastructure for them.
71+
72+
### Layer 2 — Gateway Layer (token-based)
73+
74+
External developers and partners call `api.dos.ai` directly with `dos_sk_xxx` keys.
75+
76+
| Auth | Billing | Rate limit | Use case |
77+
|------|---------|------------|----------|
78+
| `INTERNAL_API_KEY` | Skip (app layer handles) | Skip | DOSafe, DOS.AI internal services |
79+
| `dos_sk_xxx` | Per-token deduction from credits | Per-key (D1) | External developers, partners, enterprise |
80+
81+
**Why two layers?** Application-layer products know their users (Telegram chat ID, web session, etc.) and can enforce business-level quotas (free tier: 10/day, paid: 100/day). The gateway only knows API keys, which is the right abstraction for external developers.
82+
83+
---
84+
85+
## Self-Hosted GPU Fleet
86+
87+
### Current Hardware
88+
89+
| Node | GPU | VRAM | Model | Endpoint |
90+
|------|-----|------|-------|----------|
91+
| joy-pc | RTX Pro 6000 (Blackwell) | 96 GB | Qwen3.5-35B-A3B-GPTQ-Int4 (scorer) | `inference.dos.ai` |
92+
| joy-pc | RTX 5090 | 32 GB | Qwen3-8B base (observer) | `inference-ref.dos.ai` |
93+
94+
### vLLM Configuration
95+
96+
- **Scorer**: `--served-model-name dos-ai`, `--gpu-memory-utilization 0.90`, prefix caching + chunked prefill enabled
97+
- **Observer**: `--served-model-name observer`, `--enforce-eager` (CUDA graph fails on RTX 5090)
98+
- **Qwen3.5**: Thinking model — requires `chat_template_kwargs: {"enable_thinking": false}` for non-chat workloads
99+
- **Docker**: Named volume `vllm-compile-cache` for CUDA graph persistence
100+
101+
### Performance (Qwen3.5-35B-A3B-GPTQ-Int4)
102+
103+
| Metric | Value |
104+
|--------|-------|
105+
| Single request throughput | 184 tok/s |
106+
| @10 concurrent | 800 tok/s |
107+
| @50 concurrent | 3,373 tok/s |
108+
| TPOT (after warmup) | 6.6ms |
109+
| TTFT median @1 rps | 48ms |
110+
| VRAM usage | 23 GiB |
111+
| JSON compliance | 95% |
112+
| Scam detection F1 | 0.970 |
113+
114+
---
115+
116+
## LLM Fallback Chain
117+
118+
When self-hosted vLLM is unavailable, requests fall back to paid providers:
119+
120+
```
121+
1. Self-hosted vLLM (api.dos.ai → inference.dos.ai)
122+
↓ fail (timeout, 502, model not loaded)
123+
2. Alibaba Cloud DashScope International
124+
Model: qwen3.5-flash ($0.10/$0.40 per 1M tokens)
125+
↓ fail
126+
3. (Future) Google Gemini 3 Flash, OpenAI, etc.
127+
```
128+
129+
### Fallback Cost Tracking
130+
131+
When a paid provider is used, structured JSON is logged to Vercel console:
132+
133+
```json
134+
{
135+
"event": "llm_fallback_used",
136+
"provider": "qwen3.5-flash",
137+
"prompt_tokens": 4800,
138+
"completion_tokens": 180,
139+
"cost_usd": 0.000552,
140+
"entity_type": "phone"
141+
}
142+
```
143+
144+
Filter by `llm_fallback_used` in Vercel logs to monitor paid API spend.
145+
146+
User-facing billing is **not affected** by which provider served the request — it stays request-based at the application layer.
147+
148+
---
149+
150+
## API Gateway (Cloudflare Worker)
151+
152+
### Endpoint: `api.dos.ai`
153+
154+
| Route | Purpose |
155+
|-------|---------|
156+
| `POST /v1/chat/completions` | OpenAI-compatible chat API |
157+
| `GET /v1/models` | List available models |
158+
| `GET /dashboard/*` | Internal dashboard API (X-Dashboard-Secret) |
159+
160+
### Key Types
161+
162+
| Key format | Purpose | Billing |
163+
|------------|---------|---------|
164+
| `INTERNAL_API_KEY` (env secret) | DOS internal services | None (bypass) |
165+
| `dos_sk_xxx` | External developers/partners | Token-based, credits deducted |
166+
167+
### Infrastructure
168+
169+
| Component | Service | Purpose |
170+
|-----------|---------|---------|
171+
| Worker | Cloudflare Workers | Auth, routing, metering, billing |
172+
| D1 | Cloudflare D1 | API keys, rate limit log, usage log |
173+
| Supabase | `dosai` schema | Usage transactions, user settings |
174+
| DOS.Me | `public` schema | Billing accounts, credit transactions |
175+
176+
---
177+
178+
## Related Documents
179+
180+
| Document | Description |
181+
|----------|-------------|
182+
| [API_GATEWAY_PRODUCT_PLAN.md](API_GATEWAY_PRODUCT_PLAN.md) | Full product plan with phased rollout, pricing, design decisions |
183+
| [INFERENCESENSE_LIKE_ALPHA_MVP.md](INFERENCESENSE_LIKE_ALPHA_MVP.md) | GPU spare capacity routing — fleet management layer |
184+
| [API_CONTRACT_V1.md](API_CONTRACT_V1.md) | API contract for InferenceSense alpha endpoints |
185+
| [DOSafe-Architecture.md](DOSafe-Architecture.md) | DOSafe system architecture (consumer of api.dos.ai) |

0 commit comments

Comments
 (0)