diff --git a/.claude/skills/torq-developer/SKILL.md b/.claude/skills/torq-developer/SKILL.md new file mode 100644 index 000000000..7b53a3b44 --- /dev/null +++ b/.claude/skills/torq-developer/SKILL.md @@ -0,0 +1,310 @@ +--- +name: torq-developer +description: TorQ framework and kdb+/q development. TRIGGER when: file is .q or in a TorQ repo; user asks about TorQ namespaces (.servers, .gw, .sub, .lg, .timer), process.csv, tickerplant/RDB/WDB/HDB/gateway, or EOD; user writes/reviews/debugs q code; kdb+ integration with Python (PyKX), Grafana, REST, or WebSockets. SKIP: unrelated q-named tooling (e.g. Qt, Q# language). +type: skill +--- + +TorQ is the Data Intellect production framework that wraps kdb+ with process management, connection tracking, logging, timers, pub/sub, and EOD lifecycle management. The authoritative source is the TorQ framework repo (commonly `$TORQHOME`); the Finance Starter Pack is the layered reference application. + +Apply the rules below without exception. Every claim about TorQ internals cites a source file. + +--- + +# CORE PRINCIPLES + +These two principles are not duplicated by the specific rules below — everything else is covered in the lettered rules (N/C/L/H/T/S/P/M/G/Q/E/A). + +1. **Fail fast** — if a required dependency is unavailable at startup, signal an error rather than starting in a degraded state. Use `.servers.startupdepcycles` or `.servers.startupdependent` to block until dependencies are ready. (`trackservers.q`) +2. **Read the downstream process before writing any IPC call** — before writing a function call or query string that will execute on another process, read that process's source files and config to establish what it actually exposes. Never assume a function exists based on a spec description or naming convention alone. This applies to every caller type: a q process calling another q process, a Python client calling a gateway, a gateway routing to a backend. + +--- + +# CODE GENERATION RULES + +## Namespace and Structure + +- **Rule N1**: All process-specific code lives in a dedicated namespace (e.g., `\d .myproc`). Return to root at end of file with `\d .`. (`torq.q` pattern) +- **Rule N2**: Use `.proc.proctype` and `.proc.procname` to identify the current process — never hardcode process identity. (`torq.q`) +- **Rule N3**: The `parentproctype` flag loads shared code for a parent type before the child type. Use `-parentproctype wdb` for sort/sortworker processes that share wdb code. (FSP `process.csv`) +- **Rule N4**: `.api.add` every public function with signature and description. (`gateway.q:596-601`) + +## Config Variables + +- **Rule C1**: Every config variable must use the guard pattern: `myvar:@[value;\`myvar;default]`. This allows override from config files and command line. (`torq.q`) +- **Rule C2**: Config layering order (each layer overrides previous): `$KDBCONFIG/settings/default.q` → `$KDBSERVCONFIG/settings/default.q` → `$KDBAPPCONFIG/settings/default.q` → then each does `parentproctype.q` → `proctype.q` → `procname.q`. (`torq.q`) +- **Rule C3**: Any namespaced variable (`.ns.var`) can be overridden from the command line: `-ns.var value`. (`gettingstarted.md`) +- **Rule C4**: Never define config without a guard; if it is set before the config file loads it will be silently overwritten. +- **Rule C5**: The guard `@[value;\`myvar;default]` inside `\d .ns` resolves `.ns.myvar`, not root `.myvar`. To pre-set a config variable before loading (e.g. in tests or startup scripts), set the fully-qualified name: `.ns.myvar:value` — setting a root-level `myvar` will be ignored. + +## Logging + +- **Rule L1**: Standard output: `.lg.o[\`label;"message"]`. Error: `.lg.e[\`label;"message"]`. Warning: `.lg.w[\`label;"message"]`. (`torq.q`) +- **Rule L2**: `.lg.o` writes to stdout log (`out_` file). `.lg.e` writes to stderr log (`err_` file). Both write to in-memory `logmsg` table and publish via `.ps` if pubsub is active. (`torq.q`) +- **Rule L3**: Extend logging with `.lg.ext:{[loglevel;procname;label;msg] ...}` hook — this fires on every log call. (`torq-patterns.md`) +- **Rule L4**: Use `-jsonlogs` flag to switch log format to JSON (`.lg.format` is set by `torq.q` on startup). + +## Message Handlers + +- **Rule H1**: All `.z.*` overrides use `.dotz.set[\`.z.xx; newfunc]`. Never assign `.z.pc` etc. directly. (`torq.q`, `discovery.q`, `gateway.q`) +- **Rule H2**: To chain onto an existing handler: `{x@y; .myns.mypc[y]}@[value;.dotz.getcommand[\`.z.pc];{{[x]}}]`. (`gateway.q:523`) +- **Rule H3**: `.dotz.set` in FinSpace uses `.awscust.z` namespace — another reason to never bypass it. (`torq.q`) +- **Rule H4**: Message handlers loaded from `$KDBCODE/handlers/` directory. Disable all handlers with `.proc.loadhandlers:0b` in config. (`handlers.md`) + +## Timers + +- **Rule T1**: Repeating timer: `.timer.repeat[starttime;endtime;period;func;"description"]`. One-shot: `.timer.once[firetime;func;"description"]`. (`torq-patterns.md`) +- **Rule T2**: Timer modes — 0: reschedule at `T0+P` (fixed rate); 1: reschedule at `T1+P` (from fire time); 2: reschedule at `T2+P` (from completion). (`utilities.md`) +- **Rule T3**: A timer function that throws an error is removed from the timer (active set to 0b). Always wrap error-prone timer functions. (`.timer.timer` table, `cheatsheet.md`) +- **Rule T4**: Check timer is enabled: `if[not .timer.enabled; .lg.e[...]]` before registering timers. (`rdb.q:39`) + +## Table Schemas + +- **Rule S1**: Tables must have `time` as first column and `sym` as second column (for tickerplant subscription compatibility). (`SKILL.md`) +- **Rule S2**: `sym` column must carry `` `g# `` attribute in RDB in-memory tables for fast lookup: `sym:\`g#\`symbol$()`. (`database.q`, `tickerplant.q:40`) +- **Rule S3**: `upd` function must be at root namespace (not namespaced) so the tickerplant can call it. (`SKILL.md`, `tickerplant.q:34`) +- **Rule S4**: Any table that is published must have its schema defined on the receiving process. By default all schemas should be added to whatever schema file the tickerplant loads at startup (identified by the schemafile argument) +- **Rule S5**: Tables saved to HDB must be enumerated against the sym file using `.Q.en[hdbdir; table]` before writing. (`rdb.q:56`, `wdb.q`) +- **Rule S6**: After EOD writedown, re-apply attributes: functional update `![table;();0b;dict_of_attr_exprs]` each table. (`rdb.q:115`) +- **Rule S7**: All tables in the tickerplant schema file must be **unkeyed** (type 98h). The tickerplant's `.u.upd` and `.u.init` functions only handle plain tables — a keyed table (type 99h) causes startup failure. If a process needs upsert semantics for its internal state (e.g. a last-value cache), define the keyed table privately in its own namespace and publish an unkeyed flattened version (`0!`) to the tickerplant. + +## Subscriptions and Tickerplant + +- **Rule P1**: Subscribe using `.sub.subscribe[tables;syms;schema;replaylog;proc]` — do not write your own subscription protocol. The 5th arg `proc` is a dictionary row from `.sub.getsubscriptionhandles[proctype;procname;attributes]`, not a raw integer handle. Always resolve the proc dict at startup before calling subscribe. (`subscriptions.q:80`, `rdb.q:163-169`) +- **Rule P2**: Tables to ignore at EOD (not saved to disk): define in `ignorelist` variable, default `` `heartbeat`logmsg ``. (`rdb.q:18`) +- **Rule P3**: The `subfiltered` flag in rdb enables column/row filtering at subscription time. (`rdb.q:33`) + +## Connection Management + +- **Rule M1**: Declare which process types to connect to: `.servers.CONNECTIONS:\`rdb\`hdb\`gateway` (source: `trackservers.q:13`). Then call `.servers.startup[]` explicitly at the end of the load file — TorQ does NOT call this automatically; every process is responsible for calling it itself. The RDB/WDB call it inside their own startup functions. (`rdb.q:211`, `gateway.q:532`, `filealerter.q:191`) +- **Rule M2**: Get handles: `.servers.getservers[\`proctype;\`hdb;()!();1b;0b]` returns a table with `w` (handle), `procname`, `proctype`, `hpup`, `attributes`, `attribmatch`. (`trackservers.q:75-89`) +- **Rule M3**: Shortcut: `.servers.gethandlebytype[\`hdb;\`roundrobin]` returns a single handle using `roundrobin`, `any`, or `last` selection. (`trackservers.q:106`) +- **Rule M4**: `.servers.SERVERS` table columns: `procname`, `proctype`, `hpup`, `w` (handle int), `hits` (int), `startp` (timestamp), `lastp` (timestamp), `endp` (timestamp), `attributes` (dict). (`trackservers.q:10`) +- **Rule M5**: IPC type per proctype: `.servers.SOCKETTYPE:enlist[\`tickerplant]!enlist \`unix`. Options: `` `tcp`tcps`unix ``. (`conn.md`, `trackservers.q:29`) +- **Rule M6**: Password file location: `$KDBCONFIG/passwords/`. Hierarchical: `default.txt` → `proctype.txt` → `procname.txt`. (`trackservers.q:44`) +- **Rule M7**: Non-TorQ processes go in `$KDBCONFIG/settings/nontorqprocess.csv`. Enable with `.servers.TRACKNONTORQPROCESS:1b`. (`conn.md`) + +## Gateway Patterns + +- **Rule G1**: Async (deferred-sync) call: `neg[h](`.gw.asyncexec`;query;\`rdb); h[]`. The `h[]` blocks until result returns. (`gateway.q:597`) +- **Rule G2**: Async with join function and postback: `.gw.asyncexecjpt[query;servertype;joinfunction;postback;timeout]`. (`gateway.q:596`) +- **Rule G3**: Sync call: `h(`.gw.syncexec`;query;\`rdb\`hdb)`. Uses `-30!` deferred sync on kdb+≥3.6. (`gateway.q:479`) +- **Rule G4**: `servertype` argument can be: symbol list `` `rdb`hdb `` (OR — any matching type), or a dict of attributes `` enlist[`tables]!enlist enlist`trade `` (filter by attribute). (`gateway.q:392-408`) +- **Rule G5**: Gateway blocks queries during EOD reload (`.gw.eod` flag). `reloadstart` propagates timeout errors to in-flight queries; `reloadend` re-enables after HDB reload. (`gateway.q:560-577`) +- **Rule G6**: Register process attributes for routing: `update attributes:(enlist mydict) from \`.gw.servers where servertype=\`hdb`. Use `.proc.getattributes` to expose process attributes. (`gateway.q:579-587`) +- **Rule G7**: `.servers.addprocscustom` on the gateway calls `runnextquery[]` and `addserversfromconnectiontable` when new processes register. (`gateway.q:543-548`) + +## q Language Pitfalls + +- **Rule Q1**: `-x` where x is a variable name containing a numeric type will fail with a 'type error. Only use `-` when directly referencing the numeric literal e.g. `-1`, `-0.5`, `-1j`. For variables, use `neg x`. +- **Rule Q2**: Symbols containing hyphens (e.g. `` `GBP-USD ``) cannot be written with backtick syntax in q. Always cast from a string: `` `$"BTC-USD" ``. +- **Rule Q3**: `enlist x` where `x` is a symbol atom produces type 11h (symbol list) as required by typecheck functions. `enlist enlist x` produces type 0h (generic list) and will fail type checks. If you're unsure if x will always be a list, use `x:x,()` at the start of the function to ensure it's always a list (empty or not). + +## EOD Patterns + +- **Rule E1**: Add custom EOD logic by overriding `.save.postreplay:{[hdbdir;date] ...}`. This is called after all tables are written, before HDB reload. (`rdb.q:43`, `wdb.q:79`) +- **Rule E2**: Manipulate tables before EOD writedown via `.save.savedownmanipulation:enlist[\`trade]!enlist myFunc`. (`rdb.q:42`) +- **Rule E3**: `endofday` function is placed at root namespace (`endofday:.rdb.endofday`) so tickerplant can call it by name. (`rdb.q:214`) +- **Rule E4**: WDB modes: `saveandsort` (default), `save` (save only, signal sort process), `sort` (sort only). Set via `.wdb.mode`. (`wdb.q:12`) +- **Rule E5**: WDB notifies gateway via `informgateway(\`reloadstart;\`)` before sorting, `informgateway(\`reloadend;\`)` after. (`wdb.q:261`, `wdb.q:278`) +- **Rule E6**: HDB reload triggered by WDB calling `notifyhdb` which sends `` (`reload;date) `` to each HDB handle. (`rdb.q:82-83`) +- **Rule E7**: RDB with `reloadenabled:1b` does not save to disk — instead stores row counts, then WDB calls `reload` on the RDB to drop old rows. (`rdb.q:95-101`) + +## API Documentation + +- **Rule A1**: Document all public functions: `.api.add[\`.ns.func;1b;"description";"params";"return"]`. (`gateway.q:596`) +- **Rule A2**: Search functions: `.api.f\`pattern` (all), `.api.p\`pattern` (public), `.api.u\`pattern` (user-defined). (`.api.s"*pattern*"` searches function bodies.) (`utilities.md`) +- **Rule A3**: Export current config: `.api.exportconfig[\`.myns]` returns table of all variables with current values. (`utilities.md`) + +--- + +# CODE REVIEW CHECKLIST + +1. **Namespace discipline** — Does all code use `\d .ns` / `\d .` correctly? No accidental root-namespace pollution? +2. **Config guard pattern** — Every config variable uses `@[value;\`var;default]`? +3. **No raw `hopen`** — All connections go through `.servers.*` functions? +4. **`CONNECTIONS` completeness** — Every proctype passed to `.servers.gethandlebytype` or `.servers.getservers` is explicitly listed in `.servers.CONNECTIONS`? A missing entry silently produces `0Ni` handles at runtime with no error at definition time. +5. **`.dotz.set` for handlers** — No direct `.z.*` assignment? +6. **Logging** — `.lg.o`/`.lg.e`/`.lg.w` used (not `0N!`)? +7. **Timer safety** — Timer-called functions wrapped in error traps so failures don't silently remove them from the timer? +8. **Table schema** — `time` first, `sym` second, `` `g# `` on sym, `upd` at root namespace? Every table published via `upd` has a matching entry in the tickerplant's `-schemafile`? +9. **`endofday` at root** — `endofday` and `reload` assigned to root namespace so TP can call them? +10. **EOD hooks** — Custom EOD logic uses `.save.postreplay`/`.save.savedownmanipulation` rather than modifying core functions? +11. **`.api.add`** — All public functions documented? +12. **Error trapping** — `@[f;arg;{.lg.e[\`label;x]}]` or `.[f;args;{.lg.e[\`label;x]}]` used around operations that can fail? +13. **Subscription** — Uses `.sub.subscribe`, respects `ignorelist`, handles `reloadenabled` flag? +14. **Gateway routing** — `servertype` can be dict for attribute-based routing; EOD guard in place? +15. **IPC type** — Connection type (tcp/tcps/unix) configured via `.servers.SOCKETTYPE` not hardcoded? +16. **Dependency startup** — Process blocks until required connections are available using `.servers.startupdepcycles`? +17. **Credentials** — New process has a `$KDBAPPCONFIG/passwords/{proctype}.txt` (or `procname.txt`) file, AND its username appears in the `U` access list file of every process it connects to? Missing either side silently gives `'access` at connection time. +18. **IPC call verification** — For every new IPC call or query string targeting a downstream process: have you read that process's source files and config to confirm the function exists there? For a gateway: have you checked its settings file and any loaded shared code (e.g. `appconfig/settings/gateway.q`, `code/cryptofunctions/`) to distinguish functions that live on the gateway itself from those that must be routed to a backend via `.gw.syncexec`/`.gw.asyncexec`? (Core Principle 2) +19. **Staged new-process workflow** — When the change introduces a NEW process: is Stage 1 (plumbing — schemas, skeleton, `CONNECTIONS`, credentials, `process.csv` entry, subscription registration) cleanly separable from Stage 2 (feature logic)? If a single change mixes plumbing and feature logic, flag it and request Stage 1 be verified to start and connect on its own first. See PROCESS SETUP GUIDE. + +--- + +# PROCESS SETUP GUIDE + +**New processes must be added in two stages with a hard verification gate between them.** Do not write any feature/business logic in Stage 1, and do not begin Stage 2 until Stage 1 verification passes. Layering feature code onto broken plumbing produces bugs that look like logic errors but aren't — they waste hours and lead to wrong fixes. + +If you are asked to "add a process that does X", translate this into: Stage 1 first, verify, then Stage 2 for X. Do not collapse the stages to save time. + +## Stage 1 — Plumbing + +Goal: a process that starts cleanly, opens handles to every declared downstream connection, and (if it subscribes) registers with the tickerplant. **No business logic.** + +Tasks: + +1. **Schemas** — define any new published tables in the tickerplant's `-schemafile` (Rule S4). Unkeyed (Rule S7); `time` first, `sym` second with `` `g# `` (Rules S1, S2). +2. **Skeleton** (`code/processes/myproc.q`): + - Open namespace: `\d .myproc` + - Config guards on every tunable (Rule C1) + - Entry function that only logs (e.g. `run:{[] .lg.o[\`run;"stub"]}`) — no real work + - Return to root: `\d .` + - Root-level `upd` if the process subscribes (Rule S3) +3. **Connections** — `.servers.CONNECTIONS:\`typeA\`typeB\`…` listing every downstream proctype (Rule M1). Call `.servers.startup[]` at end of file. +4. **Credentials** — create `$KDBAPPCONFIG/passwords/{proctype}.txt` AND append the user to the `U` access list of every process this one will connect to (checklist item 17). +5. **Register** — add a row to `$KDBAPPCONFIG/process.csv` (column format in `torq-process-templates.md`). +6. **Subscribe (if applicable)** — block until the TP is up with `.servers.startupdepcycles`, then call `.sub.subscribe` with the proc dict from `.sub.getsubscriptionhandles` (Rule P1). + +For the skeleton template and proctype-specific templates, read `torq-process-templates.md`. + +## Stage 1 Verification — blocks Stage 2 + +Start the process and confirm **every** item below before writing a single line of feature logic: + +- [ ] Process starts: `./torq.sh start {procname}` succeeds and the PID persists (no immediate exit) +- [ ] `err_{procname}_*.log` contains no `ERR` lines after startup +- [ ] `out_{procname}_*.log` shows the expected startup/connection messages +- [ ] In qcon: `select proctype, w from .servers.SERVERS where proctype in .servers.CONNECTIONS` — every row has a non-null `w` +- [ ] If subscribing: `.sub.SUBSCRIPTIONS` shows the expected tables/syms, and on the TP `exec w from .u.w.{table}` includes this process's handle +- [ ] If publishing: the receiving process has the table schema loaded (`tables \`.` includes the new table) + +If any check fails: stop, diagnose, and fix. Do not proceed to Stage 2 to "see if it still works" — it will mask whichever Stage 1 issue is still broken. + +## Stage 2 — Feature logic + +Only after Stage 1 verification passes. Add query/timer/publish/subscription-processing logic one piece at a time, each with `.lg.o` breadcrumbs, and confirm each against the running process before adding the next. + +--- + +# DEPLOYMENT + +For env vars, deployment directory layout, `setenv.sh`, `process.csv` columns, config/code layering order, and `torq.sh` commands, read `torq-process-templates.md`. + +--- + +# DEBUGGING GUIDANCE + +## First Steps + +1. Check **error log** (`err_` file in `$KDBLOGS`) — should be empty in healthy system +2. Query **`.usage.usage`** — sorted by `timer` descending to find slow timer calls +3. Check **`.timer.timer`** — `active=0b` means the function threw an error and was removed +4. Check **`.servers.SERVERS`** — `endp` not null means a connection died +5. Check **`.clients.clients`** — frequent connect/disconnect cycles indicate client issues + +## TorQ Diagnostic Functions + +```q +.usage.usage // all queries (status "b"=before, "c"=complete, "e"=error) +select from .usage.usage where time within (start;end) +100 sublist `timer xdesc .usage.usage // slowest timer calls + +.timer.timer // scheduled timer calls; active=0b means disabled by error +.servers.SERVERS // outbound connections +.clients.clients // inbound connections + +.api.f`pattern // find functions/vars matching pattern +.api.p`pattern // public functions only +.api.s"*pattern*" // search function bodies +.api.m[] // memory usage of all variables +.api.exportconfig[`.myns] // current config values for namespace +.api.whereami[.z.s] // name of current function (useful in errors) + +.Q.w[] // workspace: used/heap/peak/wmax/mphy +.gc.run[] // force garbage collection +``` + +## Common Error Table + +| Error | Likely Cause | +|---|---| +| `'type` | Wrong type passed to function; check `-7h$x` type codes | +| `'length` | Conformability issue in vector operation | +| `'domain` | `til -1`, enum lookup failure, out-of-range cast | +| `'value` | Undefined variable or function; check namespace | +| `'wsfull` | Out of memory; call `.gc.run[]`, check `.Q.w[]` | +| `'conn` | Too many connections (pre-4.1t limit: 1022); or connection refused | +| `'timeout` | `hopen` timeout (`.servers.HOPENTIMEOUT`); or query timeout (`-T`) | +| `'access` | Auth failure or `.access` restrictions; check `.z.pw` / access list | +| `'stack` | Recursion too deep; replace with iterators | +| `'globals` | Too many global variables in function (max 8 params) | +| `'assign` | Attempt to modify a constant or read-only table | + +## Debugging Workflow + +```bash +# 1. Stop process +./torq.sh stop rdb1 + +# 2. Debug in foreground (shows full startup) +./torq.sh debug rdb1 + +# Or with error trapping to see past startup errors: +q torq.q -proctype rdb -procname rdb1 -trap -debug -load code/processes/rdb.q + +# 3. In the q session, inspect state: +q).servers.SERVERS // are connections up? +q)tables`. // what tables exist? +q).usage.usage // recent queries +``` + +## Missing Data — Debugging Protocol + +When a table is not populating and there are no obvious errors, work from the data inward — not from the framework outward. Do not start by reading subscription/distribution source. + +1. **Check live state** — qcon into the receiving process. Does the table exist? Does it have rows? Then qcon into the upstream publisher and confirm data exists there. + +2. **Call `upd` directly** — every subscriber has a root-level `upd` (or equivalent function). Call it manually with a representative row and check whether the downstream state updates as expected. + +4. **Trace logic line by line** — copy the body of the suspect function into the q session and run each statement in isolation using real values from the live state. Check intermediate results: empty tables after a filter, null handles, and wrong timestamps all become obvious immediately without needing to reason about the surrounding framework. + +5. **Check the plumbing last** — only if data exists upstream, `upd` works with test input, and the logic is sound should you look at the subscription layer: `.u.w` on the TP, `.servers.SERVERS` on the subscriber, recent errors in `.usage.usage`. + +## localtime vs Data Timestamps + +`localtime` in `process.csv` controls whether `.proc.cp[]` returns `.z.P` (local) or `.z.p` (UTC). This setting is independent of the timestamps carried in the data itself, which are determined by the feed source. + +If processes run with `localtime:1` but the feed produces UTC timestamps, any time comparison between `.proc.cp[]` and a data `time` column will be off by the local UTC offset. Staleness filters, time-windowed selects, and EOD logic are all affected. Set `localtime:0` for all processes when the feed produces UTC data. + +## Combining Out and Error Logs + +```bash +# Merge and sort both log files chronologically +sort -nk1 out_rdb1_*.log err_rdb1_*.log + +# Find last N lines before an error +sort -nk1 out_rdb1_*.log err_rdb1_*.log | grep -B 20 ERR +``` + +## Key Flags for Debugging + +- `-debug`: equivalent to `-nopi -noredirect` (interactive q session, no log redirect) +- `-trap`: catch init errors and continue +- `-stop`: halt at init error without exiting +- `-noconfig`: skip config loading (test with bare TorQ) +- `-onelog`: write all output to stdout log (easier to grep) + +--- + +# COMPANION FILES + +Only read when the current task matches — these are not auto-loaded. + +- **`torq-internals.md`** — read when debugging startup order, EOD sequence, gateway request lifecycle, or discovery protocol. +- **`torq-patterns.md`** — read when authoring new process code: namespace table, IPC/subscription patterns, caching, async helpers, error-trapping idioms. +- **`torq-process-templates.md`** — read when creating a new process, editing `process.csv`, setting up deployment, or needing a concrete template (minimal proc, feedhandler, RDB, WDB, gateway). Also holds env vars, `setenv.sh`, `torq.sh` commands, and the deployment checklist. +- **`q-language-reference.md`** — general q/kdb+ reference (not TorQ-specific). Read when hitting type errors, iterator edge cases, IPC quirks, date/time traps, or performance issues. +- **`kdb-ecosystem.md`** — read when integrating kdb+ with Python (PyKX/embedPy), Grafana, REST/HTTP, WebSockets, or C API. +``` + +--- \ No newline at end of file diff --git a/.claude/skills/torq-developer/kdb-ecosystem.md b/.claude/skills/torq-developer/kdb-ecosystem.md new file mode 100644 index 000000000..31dec5d4b --- /dev/null +++ b/.claude/skills/torq-developer/kdb-ecosystem.md @@ -0,0 +1,418 @@ +# kdb+ Ecosystem Integration Reference + +## 1. PyKX — Python/kdb+ Integration + +Source: https://code.kx.com/pykx/ + +### Modes of Operation + +PyKX supports four distinct operating modes: + +| Mode | Description | +|---|---| +| **Embedded q** | Run q code directly within the Python process | +| **IPC client** | Connect to a remote kdb+ process from Python | +| **Embedded Python (q→Python)** | Call Python from within a running q process | +| **Server mode** | PyKX exposes a Python process as a kdb+-queryable service | + +### IPC Connection Pattern + +```python +import pykx as kx + +# Connect to a running kdb+ process +with kx.SyncQConnection(host='localhost', port=5000, username='admin', password='admin') as q: + # Execute q code + result = q('select from trade where date=.z.d') + + # Call a q function + result = q('.gw.syncexec', kx.CharVector('`$last .z.x'), kx.SymbolAtom('rdb')) + + # Convert to pandas + df = result.pd() + +# Async connection +async with kx.AsyncQConnection(host='localhost', port=5000) as q: + result = await q('1+1') +``` + +### Type Mapping (Python → kdb+) + +| Python | kdb+ | PyKX class | +|---|---|---| +| `int` | long (-7h) | `kx.LongAtom` | +| `float` | float (-9h) | `kx.FloatAtom` | +| `str` | symbol (-11h) | `kx.SymbolAtom` | +| `str` | char vector (10h) | `kx.CharVector` | +| `bool` | boolean (-1h) | `kx.BooleanAtom` | +| `datetime.date` | date (-14h) | `kx.DateAtom` | +| `datetime.datetime` | timestamp (-12h) | `kx.TimestampAtom` | +| `list` | mixed list (0h) | `kx.List` | +| `numpy.ndarray` | typed vector | `kx.LongVector` etc | +| `pandas.DataFrame` | table (98h) | `kx.Table` | +| `pandas.Series` | vector | typed vector | +| `dict` | dictionary (99h) | `kx.Dictionary` | +| `None` | null | type-dependent null | + +### Embedded q Pattern (Python runs q in-process) + +```python +import pykx as kx + +# Execute q expressions +result = kx.q('1+1') # returns kx.LongAtom(2) +result = kx.q('{x+y}', 1, 2) # call with args + +# Access q namespaces +print(kx.q('.z.p')) # current timestamp + +# Convert to Python types +val = int(kx.q('42')) +df = kx.q('([]a:1 2 3; b:4 5 6)').pd() + +# Load q files +kx.q.system.load('/path/to/myfile.q') +``` + +### Known Limitations / Failure Modes + +1. **Symbol pollution**: Converting large Python string lists to kx.SymbolVector interns all strings into the q symbol table permanently — use `CharVector` for high-cardinality strings. +2. **Large table transfers**: Serialization overhead for large tables. Profile with `-8!` byte size before transfer. +3. **Attribute loss**: Attributes (`g#`, `p#` etc.) are not preserved through Python round-trips. +4. **Enumeration handling**: Enum columns are automatically de-enumerated to symbols on IPC transmission. +5. **Null handling**: q nulls map to Python `None` / pandas `NaN`; be careful with downstream type assumptions. +6. **Thread safety**: PyKX connections are not thread-safe — use one connection per thread or use connection pools. +7. **Timeout**: Default connection has no timeout — always set `timeout=` parameter in production. + +--- + +## 2. embedPy (Python within q) + +### Loading embedPy + +```q +// Load embedPy in a q session +\l p.q + +// Or via command line +q p.q myfile.q +``` + +### Calling Python from q + +```q +// Import Python modules +np:.p.import`numpy +pd:.p.import`pandas + +// Call Python functions +arr:np[`:array][1 2 3 4 5] +result:np[`:mean][arr] + +// Convert between types +pylist:.p.py 1 2 3 4 5 // q list → Python list +qlist:.p.q pylist // Python list → q list + +// Execute Python code string +.p.e "import datetime; x = datetime.date.today()" + +// Get Python variable +today:.p.get`x +``` + +### Common embedPy Patterns + +```q +// Use pandas for complex manipulations +pd:.p.import`pandas +df:pd[`:DataFrame][.Q.en[`:hdb;select from trade where date=.z.d]] +grouped:df[`:groupby]["sym"][`:agg][.p.py{`size`sum!("count";"sum")}] + +// Scikit-learn model +sklearn:.p.import`sklearn.linear_model +model:sklearn[`:LinearRegression][] +model[`:fit][features;targets] +predictions:model[`:predict][newfeatures] +``` + +--- + +## 3. REST/HTTP Integration + +Source: https://code.kx.com/q/kb/http/ + +### Implementing a REST Endpoint in kdb+ + +```q +// .z.ph — HTTP GET handler +// x = string of entire HTTP request +.z.ph:{ + // Parse the URL and query string + url:first "?" vs x; + params:(!). flip "==" vs/: "&" vs last "?" vs x; + + // Route based on URL path + path:1_ url; + result:$[ + path~"trade"; .j.j select from trade where date=.z.d; + path~"quote"; .j.j select from quote where date=.z.d; + // 404 + .h.hn["404 Not Found"; "text/plain"; "unknown endpoint: ",path] + ]; + .h.hn["200 OK"; "application/json"; result] + } + +// .z.pp — HTTP POST handler +.z.pp:{ + // x contains headers + body + body:last "\r\n\r\n" vs x; + data:.j.k body; // parse JSON body + // process data... + .h.hn["200 OK"; "application/json"; .j.j `status`msg!(`ok;"processed")] + } +``` + +### JSON Utilities + +```q +.j.j x // q value → JSON string +.j.k x // JSON string → q dict/table + +// Pitfalls: +// .j.j on a symbol list gives ["a","b"] (JSON array of strings) +// .j.j on a table gives [{"col1":val,...}, ...] +// Timestamps become strings: .j.j 2024.01.01D00:00:00 → "2024-01-01T00:00:00.000000000" +// Nulls: .j.j 0N → "null" ; .j.j 0n → "null" +// Long vs float: .j.j 42 → "42" ; .j.j 42.0 → "42.0" +``` + +### JSON Edge Cases + +```q +// Round-trip loss: JSON has no symbol type +// .j.k (.j.j `a`b`c) → ("a";"b";"c") (strings, not symbols) + +// Null handling +.j.j (0N;0n;0Nd) // "null","null","null" — type information lost + +// Large integer precision +// JavaScript JSON.parse loses precision on longs > 2^53 +// Consider string-encoding large IDs + +// Nested tables +.j.j ([]a:1 2; b:(1 2;3 4)) // nested arrays — complex round-trip +``` + +### .z.ph Security Considerations + +```q +// Default .z.ph evaluates arbitrary q — DANGEROUS in production +// Always override with restricted handler + +// Restrict by user +.z.ph:{ + if[not .z.u in `admin`readonly; .h.hn["403 Forbidden";"text/plain";"access denied"]]; + // ... safe handler + } + +// Use .z.ac for HTTP authentication (LDAP, OAuth2, OpenID Connect) +.z.ac:{[x] ...} // x = (headerdict; requestbody) +``` + +### HTTP Client (outbound requests) + +```q +// GET request +result:.Q.hg `$":http://api.example.com/data" + +// POST request +result:.Q.hp[`$":http://api.example.com/data"; "application/json"; .j.j mydata] + +// Low-level (returns bytes) +h:hopen `$":http://api.example.com" +h "GET /data HTTP/1.1\r\nHost: api.example.com\r\n\r\n" +``` + +--- + +## 4. WebSocket Integration + +Source: https://code.kx.com/q/kb/websockets/ + +### Server-Side WebSocket Handler + +```q +// Start q with port: q myfile.q -p 5000 + +// Basic echo server +.z.ws:{neg[.z.w] x} + +// JSON pub/sub pattern +.z.wo:{[h] // on WebSocket open + `wsclients upsert (h; .z.p) + } + +.z.wc:{[h] // on WebSocket close + delete from `wsclients where handle=h; + delete from `wssubs where handle=h; + } + +.z.ws:{[msg] // on message received + req:.j.k msg; + $[req[`type]~"subscribe"; + // register subscription + [`wssubs upsert (.z.w; req`tables; req`syms)]; + req[`type]~"query"; + // execute query and return result + [result:@[value;req`query;{`error`msg!(`error;x)}]; + neg[.z.w] -8! .j.j result]; // -8! = serialize to bytes for WS + // unknown + neg[.z.w] -8! .j.j `error`msg!(`unknown;"unknown message type") + ] + } + +// Publish to all subscribers +pubws:{[table;data] + subs:select handle from wssubs where table in tables; + msg:.j.j (`type`table`data!(`update;table;data)); + (neg each exec handle from subs) @\: -8! msg; + } +``` + +### WebSocket Message Format + +```q +// Messages arrive as: +// - byte vectors (-8h type): serialized kdb+ (use -9! to deserialize) +// - char vectors (10h type): text/JSON (use .j.k to parse) + +.z.ws:{[x] + msg:$[-8h=type x; -9!x; .j.k x]; // handle both formats + ... + } +``` + +### Client-Side WebSocket (kdb+ as client, v3.2+) + +```q +// Connect to WebSocket server +// Returns (handle; HTTP_response_string) +r:(`$":ws://localhost:5001/stream")"GET / HTTP/1.1\r\nHost: localhost:5001\r\n\r\n" +h:r 0 + +// Send message +neg[h] .j.j `type`query!(`subscribe;"trade") + +// Receive: set .z.ws before connecting +.z.ws:{[x] 0N! "Received: ", string x} +``` + +### TorQ WebSocket Integration + +TorQ wraps `.z.ws` via `.dotz.set` if it's already defined (gateway.q line 528): +```q +if[@[{value x;1b};`.z.ws;{0b}]; + .dotz.set[`.z.ws;{.gw.pgs[.z.w;0b]; x@y} value .dotz.getcommand[`.z.ws]]]; +``` +This tracks WebSocket connections as async clients in `.gw.call`. + +--- + +## 5. Grafana / AquaQ kdb+ Datasource + +### Query Contract + +The AquaQ kdb+ Grafana datasource (plugin ID: `aquaqanalytics-kdbbackend-datasource`) connects directly to a kdb+ process via IPC and calls a configured q function. + +Standard patterns for Grafana-compatible kdb+ functions: + +```q +// Time-series query: returns table with `time` and value columns +// Grafana passes: (starttime; endtime; interval; params) +grafanaTimeseries:{[starttime;endtime;interval;params] + select time, price from trade + where date within `date$(starttime;endtime), + sym=params`sym + } + +// Table query: returns plain table +grafanaTable:{[starttime;endtime;interval;params] + select sym, avg price, sum size + from trade + where date within `date$(starttime;endtime) + by sym + } +``` + +### Function Contract Requirements + +1. Must return a table +2. For time-series panels: table must have a `time` column (timestamp type) +3. For table panels: any column structure +4. Function must handle null params gracefully + +### Exposing through TorQ Gateway + +```q +// In gateway-accessible process (HDB or RDB) +.proc.getattributes:{[] `tables!enlist tables`.} + +// Grafana-facing query function on HDB +grafanaQuery:{[starttime;endtime;interval;params] + tbl:params`table; + if[not tbl in tables`.; '"table not found: ",string tbl]; + ?[tbl; + enlist (within; `date; enlist `date$(starttime;endtime)); + 0b; + ()] + } + +// Call through gateway (from Grafana → Gateway → HDB) +// Configure datasource to call: .gw.syncexec[("grafanaQuery";starttime;endtime;interval;params);`hdb] +``` + +--- + +## 6. C API / C Extensions + +### Use Cases + +- High-performance feedhandlers (C++ with kdb+ C API) +- Custom compression/decompression +- Integration with low-latency market data feeds (e.g., Solace, LBM) +- FPGA/hardware interface + +### Basic Pattern + +```c +// Link against k.h from code.kx.com +#include "k.h" + +// Open connection +I handle = khpun("localhost", 5000, "user:pass", 1000); + +// Execute query +K result = k(handle, "{x+y}", ki(1), ki(2), (K)0); + +// Check for error +if(!result || result->t == -128) { /* error */ } + +// Serialize/deserialize +K serialized = b9(-1, result); // serialize +K deserialized = d9(serialized); // deserialize + +// Decrement reference count +r0(result); +``` + +### When to Use C API vs PyKX + +| Use Case | Recommendation | +|---|---| +| Analytics/data science | PyKX | +| Scripting/automation | PyKX or q | +| Low-latency feedhandler | C API | +| Production system integration | q IPC or PyKX | +| Grafana/monitoring | HTTP/.z.ph or Grafana plugin | +| Bulk data loading | q loader or PyKX | +``` \ No newline at end of file diff --git a/.claude/skills/torq-developer/q-language-reference.md b/.claude/skills/torq-developer/q-language-reference.md new file mode 100644 index 000000000..c45283801 --- /dev/null +++ b/.claude/skills/torq-developer/q-language-reference.md @@ -0,0 +1,534 @@ +# q/kdb+ Language Reference — Edge Cases and Developer Traps + +## 1. Type System + +### Type Code Table + +| Type# | Char | Name | Size | Null | Infinity | Example | +|---|---|---|---|---|---|---| +| -1h | b | boolean | 1 | `0b` | — | `1b` | +| -4h | x | byte | 1 | `0x00` | — | `0xff` | +| -5h | h | short | 2 | `0Nh` | `0Wh` | `42h` | +| -6h | i | int | 4 | `0Ni` | `0Wi` | `42i` | +| -7h | j | long | 8 | `0Nj` / `0N` | `0Wj` / `0W` | `42` | +| -8h | e | real | 4 | `0Ne` | `0We` | `1.5e` | +| -9h | f | float | 8 | `0n` | `0w` | `1.5` | +| -10h | c | char | 1 | `" "` | — | `"a"` | +| -11h | s | symbol | * | `` ` `` | — | `` `sym `` | +| -12h | p | timestamp | 8 | `0Np` | `0Wp` | `2024.01.01D00:00:00` | +| -13h | m | month | 4 | `0Nm` | `0Wm` | `2024.01m` | +| -14h | d | date | 4 | `0Nd` | `0Wd` | `2024.01.01` | +| -15h | z | datetime | 8 | `0Nz` | `0Wz` | `2024.01.01T00:00:00` | +| -16h | n | timespan | 8 | `0Nn` | `0Wn` | `00:01:00.000000000` | +| -17h | u | minute | 4 | `0Nu` | `0Wu` | `00:01` | +| -18h | v | second | 4 | `0Nv` | `0Wv` | `00:01:00` | +| -19h | t | time | 4 | `0Nt` | `0Wt` | `00:01:00.000` | +| 0h | — | mixed list | — | — | — | `(1;2.0;"a")` | +| 10h | — | char vector (string) | — | — | — | `"hello"` | +| 11h | — | symbol list | — | — | — | `` `a`b`c `` | +| 20-76h | — | enumeration | — | — | — | `` `sym$`x `` | +| 98h | — | table | — | — | — | `([]a:1 2 3)` | +| 99h | — | dictionary | — | — | — | `` `a`b!1 2 `` | +| 100h+ | — | function/lambda | — | — | — | `{x+y}` | + +### Type Detection Traps + +**Trap 1 — Atom vs vector type codes:** +```q +q)type 42 // -7h (negative = atom) +-7h +q)type enlist 42 // 7h (positive = vector) +7h +q)type 42 43 // 7h (vector) +7h +``` + +**Trap 2 — `type` on empty lists:** +```q +q)type () // 0h (generic empty list) +0h +q)type `int$() // 6h (typed empty list) +6h +q)count `int$() // 0 (but has a type!) +0 +``` + +**Trap 3 — Mixed list promotion:** +```q +q)type 1 2 3 // 7h (long vector, NOT mixed) +7h +q)type 1 2 3.0 // 9h (promoted to float — 3.0 causes promotion) +9h +q)type (1;2.0) // 0h (mixed: int and float don't auto-promote in parens) +0h +``` + +**Trap 4 — Char vector vs list of chars:** +```q +q)type "abc" // 10h (string = char vector) +10h +q)type ("a";"b";"c") // 0h (generic list of chars, NOT a string) +0h +``` + +**Trap 5 — Symbol interning:** +Symbols are interned — once created they persist in the symbol table for the process lifetime. Never create symbols dynamically from high-cardinality data (e.g., UUIDs). Use strings or enumerations instead. +```q +// DANGEROUS: creates millions of permanent symbols +sym:`$string each til 1000000 + +// SAFE: use string +str:string each til 1000000 +``` + +**Trap 6 — Enumeration types:** +An enumerated column has type 20h-76h (depends on which domain). After loading from disk, sym columns are enums. `value enum_col` gives symbol list. `enum_col~`mysym` works correctly. + +**Trap 7 — Null comparisons:** +```q +q)0N = 0N // 0b! (null != null in q, unlike SQL IS NULL) +0b +q)null 0N // 1b (use null function) +1b +q)null each (0N;1;0Ni;0n) // 1010b +``` + +### Casting + +```q +// Numeric upcast (safe) +`float$42 // 42f +`long$42h // 42 + +// Downcast (truncates, may overflow) +`int$2147483648 // 0Ni (overflow wraps to null!) +`short$40000 // 0Nh (overflow) + +// String to symbol +`$"hello" // `hello + +// Type code cast +9h$3 // 3f (cast using type code) +-7h$3.7 // 3 (truncates) + +// Date arithmetic: dates are ints from 2000.01.01 +`int$2000.01.01 // 0 +`int$2000.01.02 // 1 +2000.01.01+1 // 2000.01.02 +``` + +--- + +## 2. Iterators and Adverbs — Edge Cases + +### Each (`'`) + +```q +q)f:{x+1} +q)f each 1 2 3 // 2 3 4 (atomic, same as f 1 2 3 for atomic f) +q)f'[1 2 3] // 2 3 4 (adverb form) + +// Each on a binary function requires two matching-length lists +q){x+y}'[1 2 3; 10 20 30] // 11 22 33 + +// Each-right and Each-left for cross-application +q)1 2 3 +/: 10 20 // (11 12 13; 21 22 23) — add each right to each left list +q)1 2 3 +\: 10 20 // (11 21; 12 22; 13 23) — each left applied to each right +``` + +**Trap — each on atomic functions is redundant but harmless:** +```q +q)neg each 1 2 3 // -1 -2 -3 (same as neg 1 2 3) +``` + +**Trap — each on string-atomic functions:** +```q +q)upper "hello" // "HELLO" (operates on entire string) +q)upper each "hello" // "HELLO" (same — upper is string-atomic) +q)upper each ("hello";"world") // ("HELLO";"WORLD") (list of strings) +``` + +### Over (`/`) and Scan (`\`) + +```q +// Over (fold/reduce) +q)0 +/ 1 2 3 4 5 // 15 +q)(+/) 1 2 3 4 5 // 15 (same, unary projection) + +// Scan (all intermediate results) +q)0 +\ 1 2 3 4 5 // 0 1 3 6 10 15 +q)(+\) 1 2 3 4 5 // 1 3 6 10 15 (no seed — starts from first element) + +// Convergence form (iterate until stable) +q){x*x}/ 0.5 // 0 (converges to 0) + +// N-times form +q)2 {x*2}/ 1 // 4 (apply 2 times: 1→2→4) + +// Until condition form +q){x<1000} {x*2}/ 1 // 1024 (apply while condition is false) +``` + +**Trap — over with seed vs without:** +```q +q)(+/) 1 2 3 // 6 (no seed: uses first element as seed) +q)0 +/ 1 2 3 // 6 (explicit seed) +q)1 +/ 1 2 3 // 7 (seed=1, result is 1+1+2+3=7) +``` + +**Trap — scan returns INCLUDING seed:** +```q +q)0 +\ 1 2 3 // 0 1 3 6 (includes seed 0) +q)(+\) 1 2 3 // 1 3 6 (no seed, starts accumulation from element 1) +``` + +### Each-Prior (`':`) + +```q +q)(-':) 1 3 6 10 // 1 2 3 4 (differences, first element treated as 0 prior) +q)1 -': 1 3 6 10 // 0 2 3 4 (explicit prior seed of 1) +``` + +### Parallel Each (`peach`) + +```q +// Requires slave threads: q myfile.q -s 4 +q){system"sleep 1";x} peach 1 2 3 4 // runs in parallel on 4 threads +``` + +**Trap — peach shares nothing, beware global state:** +```q +// Global modification in peach workers is not thread-safe +// Each worker has a copy of the process state at fork time +// Results are merged back; side effects on globals are lost +``` + +--- + +## 3. IPC Edge Cases + +### Handle Arithmetic + +```q +h:hopen `:host:5000 // positive handle (sync) +neg[h] // negative handle (async) +h (`.proc.procname;`) // sync call +neg[h] (`.proc.procname;`) // async (fire and forget) +neg[h] (::) // flush async queue (forces send) +h (::) // sync barrier (ensures prior async messages processed) +``` + +**Trap — deferred sync pattern:** +```q +// Server: gateway sends async, then blocks waiting for result +neg[h] (`.gw.asyncexec; query; `rdb) +result:h[] // block on handle until server sends back result +// This is "deferred synchronous" — server can process query async +``` + +**Trap — connection limit:** +- Pre-4.1t (2023.09.15): max 1022 connections. Exceeding gives `'conn` error. +- Post-4.1t: limit removed. + +**Trap — interrupted sync request:** +- If a process receives `kill -s INT` while a sync query is blocking, subsequent IPC attempts on that handle fail with "Bad file descriptor". +- Always `hclose` the handle and reopen. + +**Trap — message ordering:** +- All messages on a single handle are sequential. +- A sync call guarantees all prior async messages on that handle have been sent. +- To explicitly flush: `neg[h] (::)` then `h (::)`. + +**Trap — compression:** +- Automatic for messages >2000 bytes when not on localhost/UDS, and compressed size <50% of original. +- Do NOT rely on enumerations in IPC — they are automatically converted to values before transmission. + +**Trap — `hopen` timeout:** +- Default `hopen` blocks forever. +- TorQ uses `(hopen x; timeout)` form with `.servers.HOPENTIMEOUT` (default 2000ms). + +**Trap — negative handle send fails silently:** +```q +// neg[h] sends asynchronously — if it fails, you get no error here +neg[h] "invalid_expression" // no error raised in caller process +// The receiving process gets an error; use .z.ps error handling on server side +``` + +### Serialization + +```q +-8! x // serialize (convert to bytes) +-9! x // deserialize (bytes to q value) +// Use for WebSocket JSON frames or file persistence +``` + +--- + +## 4. Error Handling Scoping + +### Basic Trap + +```q +// Two-arg trap: function and error handler +@[f; x; handler] // equivalent to: @[f[x]; handler] +.[f; (x;y); handler] // binary trap (multiarg) + +// Error handler receives error string +@[{1+`a}; ::; {0N! "caught: ",x}] // prints "caught: type" + +// Return default on error +result:@[{1+`a}; ::; {0N}] // returns 0N (long null) on error +``` + +**Trap — scoping of variables in trap:** +```q +// Variables modified inside trap handler ARE visible outside if assigned globally +a:1; +@[{a::2; `err}; ::; {a::3}]; +a // 3 — the error handler ran and set a to 3 + // a::2 ran before the error, then error rolled back to handler scope +``` + +**Trap — nested errors and stack:** +```q +// Error signals propagate up the call stack +// Only the innermost @[...] catches the error +// Outer traps don't see errors caught by inner traps +f:{@[{1+`a};::;{"inner: ",x}]} +@[f;::;{"outer: ",x}] // returns "inner: type" — outer trap never fires +``` + +**Trap — signal with `'`:** +```q +'`myerror // signal symbol error +'"my error string" // signal string error +// To re-signal after catching: +@[dangerousfn; arg; {'x}] // re-raise any caught error +``` + +**Trap — exit and .z.exit:** +```q +.z.exit:{[exitcode] .lg.o[`exit;"process exiting with code ",string exitcode]} +exit 0 // clean exit; triggers .z.exit +exit 1 // error exit +``` + +--- + +## 5. Performance Traps + +**Trap 1 — `select` vs functional form:** +```q +// String parse of select is slower (parses every time) +value "select from trade where date=2024.01.01" // avoid in hot paths + +// Functional form is faster (pre-parsed) +?[`trade; enlist(=;`date;2024.01.01); 0b; ()] +``` + +**Trap 2 — Attributes matter enormously:** +```q +// Without attribute: linear scan O(n) +select from trade where sym=`AAPL // slow on large table + +// With `g# attribute: hash lookup O(1) amortized +// With `p# attribute: binary search O(log n) +// Apply at load: update sym:`g#sym from `trade +``` + +**Trap 3 — Column extraction vs table query:** +```q +// Slow: full table scan for one column +exec sym from trade + +// Fast if you need just one column: +trade`sym // direct column access, no copy +``` + +**Trap 4 — `count` before expensive operations:** +```q +// Always check count before costly operations +if[count r:select from trade where ...; + // expensive work + ] +``` + +**Trap 5 — String operations are slow on symbols:** +```q +// Slow: convert to string then compare +select from t where string[sym] like "AAPL*" + +// Fast: symbol list membership +select from t where sym in `AAPL`AAPLV`AAPLW +``` + +**Trap 6 — Global modification in loops:** +```q +// Slow: extending global list in loop (reallocates each time) +r:(); +do[1000; r,:enlist somevalue[]]; + +// Fast: build list then assign once +r:somevalue[] each til 1000 +``` + +**Trap 7 — `.Q.gc[]` timing:** +```q +// GC pauses. In latency-sensitive processes, set g:0 (deferred GC) +// and call .gc.run[] (TorQ wrapper for .Q.gc[]) at safe points (post-EOD) +``` + +**Trap 8 — `peach` overhead:** +```q +// peach has serialization overhead — only worth it for large/slow operations +// For small fast operations, each is faster than peach +``` + +--- + +## 6. Date and Time Traps + +**Trap 1 — Date arithmetic types:** +```q +q)2024.01.01 + 1 // 2024.01.02 (int+date=date) +q)2024.01.01 + 1.0 // type error! (float+date not allowed) +q)2024.01.02 - 2024.01.01 // 1 (date-date=int, not date) +``` + +**Trap 2 — Timestamp vs datetime:** +```q +// .z.p = timestamp (nanosecond precision, UTC) +// .z.P = timestamp (nanosecond precision, local time) +// .z.z = datetime (float, millisecond precision, UTC) — DEPRECATED in favour of .z.p +// TorQ: use .proc.cp[] which returns .z.p or .z.P depending on -localtime flag +``` + +**Trap 3 — Date extraction:** +```q +q)`date$.z.p // date part of timestamp +q)`time$.z.p // time part (as time type) +q)`second$.z.p // seconds since midnight (as second type) +q)"d"$2024.01.01 // same date (explicit cast) +``` + +**Trap 4 — EOD time calculation:** +```q +// .eodtime.nextroll gives next EOD roll time as timestamp +// TorQ timer set at: .eodtime.nextroll - 00:01 (one minute before) +// Adjusted for timezone offset between local and UTC +``` + +**Trap 5 — Time arithmetic:** +```q +q)12:00:00.000 + 1 // type error — can't add int to time +q)12:00:00.000 + 00:01 // 12:01:00.000 (time + minute = time) +q)12:00:00.000 + 0D00:01 // type error — timespan ≠ minute +q)12:00:00.000 + 00:01:00.000000000 // 12:01:00.000 (time + timespan) +``` + +**Trap 6 — Timezone handling:** +```q +// .tz.t = timezone table (loaded from tz.csv) +// .tz.lg = local to GMT +// .tz.gl = GMT to local +// In TorQ: .eodtime.dailyadj adjusts for DST changes +``` + +**Trap 7 — Timestamp null vs date null:** +```q +q)null 0Np // 1b (timestamp null) +q)null 0Nd // 1b (date null) +q)0Np = 0Np // 0b (null != null — always use null[] function) +``` + +--- + +## 7. Namespace and Context Traps + +**Trap 1 — `\d` persists for rest of file:** +```q +\d .myns +// ALL definitions here are in .myns +myfunc:{x+1} // becomes .myns.myfunc + +\d . +// Back to root +otherfunc:{x+1} // becomes .otherfunc (root namespace) +``` + +**Trap 2 — Backtick lookup in namespaces:** +```q +\d .myns +x:42 +f:{value `x} // looks up `x in current context = .myns.x = 42 +g:{.myns.x} // explicit — always works +``` + +**Trap 3 — `.` (root) vs `` `. `` (root namespace):** +```q +tables`. // tables in root namespace +value `. // all variables in root namespace +.Q.f[;2] // .Q namespace function +``` + +**Trap 4 — Function local variables shadow globals:** +```q +a:10 +f:{a:20; a} // local a=20, global a unchanged +f[] // 20 +a // 10 (unchanged) + +// Use :: to modify global +g:{a::20} +g[] +a // 20 (modified) +``` + +**Trap 5 — Max 8 parameters per function:** +```q +{[a;b;c;d;e;f;g;h] ...} // OK (8 params max) +{[a;b;c;d;e;f;g;h;i] ...} // 'params error +// Workaround: pass dictionary as single arg +{[args] args[`a]+args[`b]} [`a`b!1 2] +``` + +--- + +## 8. Common q Idioms Reference + +```q +// Safe value with default +@[value;`var;default] // get var or default if undefined +@[f; arg; {defaultval}] // call f with error trap + +// Conditional assignment (only if not already set) +myvar:@[value;`myvar;42] + +// Null-safe operations +42^0N // 42 (fill null with 42) +0N^42 // 42 (null^non-null = non-null... wait: ^ fills LEFT nulls) +0^0N // 0 (fill null 0N with 0) + +// In-place table update +update col:value from `mytable where condition + +// Functional select +?[`t; where_clauses; by_clause; column_dict] + +// Functional update +![`t; where_clauses; 0b; column_update_dict] + +// Column extraction from unkeyed table (returns value vector) +t`col1 // `a`b`c — column as a list +// Note: does NOT work on keyed tables; use (0!t)`col1 to unkey first + +// Safe dictionary merge (right wins) +d1,d2 + +// String formatting +"result: ",(string 42) // "result: 42" +.Q.s1 value // format any value as string (like 0N! but no print) +``` +``` + +--- \ No newline at end of file diff --git a/.claude/skills/torq-developer/torq-internals.md b/.claude/skills/torq-developer/torq-internals.md new file mode 100644 index 000000000..bc81adf2c --- /dev/null +++ b/.claude/skills/torq-developer/torq-internals.md @@ -0,0 +1,457 @@ +# TorQ Internals Deep Reference + +## 1. torq.q Startup Sequence + +Source: `torq.q` (in the TorQ framework repo) + +### Stage 1: Environment and `.proc` Namespace + +torq.q begins by entering `\d .proc` and defining: +- `.proc.loaded:0b` — set to `1b` after full initialisation +- `.proc.initialised:0b` — set to `1b` after `.proc.initlist` runs +- `.proc.initlist:()` — list of `{[]}` functions to run at end of startup +- `.proc.addinitlist:{[f] .proc.initlist,:enlist f}` — register init callbacks + +**Startup flags read from `.z.x` via `.Q.opt`:** + +| Flag | Effect | +|---|---| +| `-proctype` | Sets `.proc.proctype` | +| `-procname` | Sets `.proc.procname` | +| `-parentproctype` | Sets `.proc.parentproctype` | +| `-procfile` | Override process.csv path | +| `-localtime` | Switch `cp`/`cd`/`ct` from UTC to local time | +| `-trap` | Catch init errors, continue | +| `-stop` | Halt at init error, don't exit | +| `-debug` | `-nopi -noredirect` | +| `-noredirect` | Don't redirect stdout/stderr to log files | +| `-noconfig` | Skip config loading | +| `-nopi` | Reset `.z.pi` to default | +| `-jsonlogs` | Switch log format to JSON | +| `-test` | Unit test mode | +| `-onelog` | All output to stdout log | + +### Stage 2: Logging Setup (`.lg` Namespace) + +``` +.lg.outmap — dict of log level → output handle (default: 1 for stdout) +.lg.pubmap — dict of log level → whether to publish via .ps +.lg.format — log line formatter (plain or JSON depending on -jsonlogs) +.lg.publish — publishes to logmsg table via .ps.publish if pubsub active +.lg.l — low-level log function +.lg.o — info (INF) log +.lg.e — error (ERR) log +.lg.w — warning (WRN) log +.lg.ext — empty hook; override to extend all logging +``` + +### Stage 3: Environment Variable Substitution + +`.rmvr.removeenvvar` substitutes `{ENV_VAR}` patterns in strings read from process.csv. This is how `{KDBBASEPORT}+1` works in port columns. + +### Stage 4: Process Identification + +Reads `process.csv` via `readprocfile`. Identifies itself by matching `host` + `port` in the CSV, OR uses explicit `-proctype`/`-procname` flags. If neither works, process exits with error. + +### Stage 5: Log File Redirection + +stdout and stderr redirected to timestamped files in `$KDBLOGS`: +- `out_{proctype}_{procname}_{timestamp}.log` +- `err_{proctype}_{procname}_{timestamp}.log` +- `usage_{proctype}_{procname}_{timestamp}.log` + +Aliases created without timestamp suffix (unless `-noredirectalias`). + +### Stage 6: Config Loading + +For each config root (`KDBCONFIG`, `KDBSERVCONFIG`, `KDBAPPCONFIG`), in order: +1. `settings/default.q` +2. `settings/{parentproctype}.q` (if parentproctype set) +3. `settings/{proctype}.q` +4. `settings/{procname}.q` + +Files that don't exist are silently skipped. + +### Stage 7: Code Loading + +`reloadallcode` function iterates code roots (`KDBCODE`, `KDBSERVCODE`, `KDBAPPCODE`) and for each loads: +1. `common/` — shared utilities +2. `{parentproctype}/` — parent type code +3. `{proctype}/` — process type specific code +4. `{procname}/` — process name specific code +5. `handlers/` — message handler customisations + +Then loads any `-load` / `-loaddir` files specified on command line. + +### Stage 8: Pubsub and Servers Initialisation + +1. `.ps.initialise[]` — initialise publish/subscribe system +2. `.servers.startup[]` — read process.csv, connect to discovery, make initial connections + +### Stage 9: Init Callbacks and Finalisation + +1. Runs each function in `.proc.initlist` +2. Sets `.proc.initialised:1b` +3. Sets `.proc.loaded:1b` +4. If `-test` flag: runs unit tests and exits + +--- + +## 2. Gateway Internals + +Source: `code/processes/gateway.q` + +### Key Tables + +```q +// Keyed on queryid: tracks pending queries +queryqueue:([queryid:`int$()] clienth:`int$(); query:(); servertype:(); + queryattributes:(); joinfunction:(); postback:(); + timeout:`timespan$(); sync:`boolean$(); + senttime:`timestamp$(); returntime:`timestamp$()) + +// Tracks query results from backend servers +results:([]queryid:`int$(); serverid:`int$(); result:()) + +// Backend server registry (keyed on serverid) +servers:([serverid:`int$()] handle:`int$(); servertype:`symbol$(); + active:`boolean$(); inuse:`boolean$(); usage:`timespan$(); + attributes:()) + +// Connected clients +clients:([]handle:`int$(); connecttime:`timestamp$()) +``` + +### Request Lifecycle (Async Path) + +1. **Client calls** `neg[gw_handle](`.gw.asyncexecjpts`;query;servertype;joinf;postback;timeout;sync)` +2. **`asyncexecjpts`** validates: correct function type (sync vs async), permissions, servertype resolution +3. **`addquerytimeout`** inserts row into `queryqueue` +4. **`runnextquery`** → `runquery` checks if any queries in queue can be served +5. **`availableserverstable`** finds servers not `inuse` that match required servertype/attributes +6. **`sendquerytoserver`** sends async: `(neg handles)@\:(serverexecute;queryid;query)` +7. **Backend executes** `serverexecute[queryid;query]` → sends result back: `neg[.z.w](`.gw.addserverresult`;queryid;result)` +8. **`addserverresult`** / **`addservererror`** stores result row +9. **`checkresults`** fires when all parts of a query are complete: applies join function +10. **`sendclientreply`** sends result: `-30![handle]` for sync (deferred-sync), `neg[handle]` for async with postback + +### Public API Functions + +```q +// [query; servertype; joinfunction; postback; timeout; sync] +// Most general form. sync=1b means treat as sync (deferred-sync pattern) +asyncexecjpts:{[query;servertype;joinfunction;postback;timeout;sync] ...} + +// [query; servertype; joinfunction; postback; timeout] +// Async version (sync=0b projection of asyncexecjpts) +asyncexecjpt:asyncexecjpts[;;;;;0b] + +// [query; servertype] +// Deferred-sync: client sends async, blocks on handle. join=raze, no postback, no timeout +asyncexec:asyncexecjpt[;;raze;();0Wn] + +// [query; servertype; joinfunction; timeout] +// Uses -30! deferred sync on kdb+>=3.6; falls back to syncexecjpre36 on older +syncexecjt:{[query;servertype;joinfunction;timeout] ...} + +// [query; servertype; joinfunction] +// syncexecjpre36: send async to all backends, flush, then block on each handle +syncexecjpre36:{[query;servertype;joinfunction] ...} + +// Aliases chosen at load time based on .z.K +syncexecj:$[.z.K<3.6; syncexecjpre36; syncexecjt[;;;0Wn]] +syncexec:$[.z.K<3.6; syncexecjpre36[;;raze]; syncexecjt[;;raze;0Wn]] +``` + +### Server Routing Logic + +`getserverids[servertype]` resolves the servertype argument: +- If symbol list: looks up matching server IDs in `.gw.servers` +- If dict (attributes): calls `getserversindependent` which finds minimum set of servers needed to cover all attribute requirements independently + +`getserversindependent[req;att;besteffort]` uses a greedy coverage algorithm — sorts servers by how many requirements they satisfy, removes redundant overlapping servers. + +### EOD Handling + +```q +// Called by WDB at EOD start +reloadstart:{ + .gw.seteod[1b]; // block new multi-server queries + // Find queries not yet returned that span multiple servers + qids:exec queryid from .gw.queryqueue where ...; + // Send error to each affected client + .gw.sendclientreply[;errorprefix,"query did not return prior to eod reload";0b] each qids; + .gw.finishquery[qids;1b;0Ni]; + } + +// Called by WDB when HDB reload is complete +reloadend:{ + .gw.seteod[0b]; // re-enable queries + // Refresh attributes from all connected servers + setattributes .' flip value flip select procname,proctype,...attributes... from .servers.SERVERS; + .gw.runnextquery[]; // flush any queued queries + } +``` + +### Handler Setup + +```q +// gateway.q lines 523-528 +.dotz.set[`.z.pc; {x@y; .gw.pc[y]} @ [value;.dotz.getcommand[`.z.pc];{{[x]}}]]; +.dotz.set[`.z.po; {x@y; .gw.po[y]} @ [value;.dotz.getcommand[`.z.po];{{[x]}}]]; +.dotz.set[`.z.pg; {.gw.pgs[.z.w;1b]; x@y} @ [...]]; // 1b = sync +.dotz.set[`.z.ps; {.gw.pgs[.z.w;0b]; x@y} @ [...]]; // 0b = async +// Also wraps .z.ws if already defined +``` + +`pgs[handle;issync]` updates `.gw.call` dict — used by `asyncexecjpts` and `syncexecjpre36` to verify the correct function is used for the call type. + +### Timer Functions (set up at end of gateway.q) + +```q +.timer.repeat[...;0D00:05; (`.gw.removequeries;.gw.querykeeptime); "Remove old queries"] +.timer.repeat[...;0D00:00:05;(`.gw.checktimeout;`); "Timeout queries"] +.timer.repeat[...;0D00:05; (`.gw.removeinactive;.gw.clearinactivetime); "Remove inactive"] +``` + +### Config Variables + +```q +.gw.synccallsallowed:0b // allow sync calls (default off for performance) +.gw.querykeeptime:0D00:30 // how long to keep completed queries in queue +.gw.errorprefix:"error: " // prefix on error messages returned to clients +.gw.permissioned:0b // require .pm.allowed check +.gw.clearinactivetime:0D01:00 // remove inactive client records after 1hr +``` + +--- + +## 3. Process Discovery Protocol + +Source: `code/processes/discovery.q`, `code/handlers/trackservers.q` + +### Registration Sequence + +**At every process startup** (`trackservers.q:startup`): + +1. Read `process.csv` → `procstab` +2. If `DISCOVERYREGISTER` or `CONNECTIONSFROMDISCOVERY`: register discovery entries, call `retrydiscovery[]` +3. `retrydiscovery[]` opens connections to all discovery services in the SERVERS table, then sends async `(`..register;\`)` to each +4. If `CONNECTIONSFROMDISCOVERY`: call `registerfromdiscovery[CONNECTIONS;0b]` → queries discovery for known process types +5. If not `CONNECTIONSFROMDISCOVERY`: call `register[procs;proctype;0b]` for each required type from process.csv + +**On discovery service side** (`discovery.q:register`): + +```q +register:{ + .servers.addw .z.w; // add caller to SERVERS table (calls addhw[`]) + // De-duplicate: close old handle if same hpup re-registers + if[count toclose:...]; + // Publish update to all subscribed processes + new:select proctype,procname,hpup,attributes from .servers.SERVERS where w=.z.w; + (neg ((where ...)inter key .z.W) except .z.w)@\:(`.servers.procupdate;new); + } +``` + +**`addhw[hpup;handle]`** (`trackservers.q:139`): +1. Calls `.servers.getdetails[]` on the remote process to get `procname`, `proctype`, `port`, `attributes` +2. Inserts row into `.servers.SERVERS` + +### `.servers.SERVERS` Table Schema + +``` +Column Type Description +----------- ----------- ------------------------------------------- +procname symbol Process name (e.g., `rdb1) +proctype symbol Process type (e.g., `rdb) +hpup symbol Host:port connection string (e.g., `:host:5011) +w int Open handle (0Ni if disconnected) +hits int Number of times handle used via updatestats +startp timestamp Time handle was first opened (0Np if not connected) +lastp timestamp Time handle was last used +endp timestamp Time handle was closed (0Np if still active) +attributes dict Process-reported attributes dict from .proc.getattributes[] +``` + +Source: `trackservers.q:10` + +### Reconnection Logic + +Two timer-driven retry mechanisms: + +```q +// Retry all dead non-discovery connections every RETRY interval (default 5 min) +if[RETRY > 0; .timer.repeat[...;RETRY;(`.servers.retry;`);"Attempt reconnections..."]] + +// Retry discovery specifically every DISCOVERYRETRY interval (default 5 min) +if[DISCOVERYRETRY > 0; .timer.repeat[...;DISCOVERYRETRY;(`.servers.retrydiscovery;`);...]] +``` + +`retry[]` calls `retryrows` on all rows where handle is dead and proctype != `` `discovery ``. +`retrydiscovery[]` handles discovery specifically, then calls `registerfromdiscovery` to refresh known processes. + +### When Discovery is Unavailable + +- If `DISCOVERYREGISTER:0b` and `CONNECTIONSFROMDISCOVERY:0b`: process reads connections statically from process.csv only — fully offline mode +- If discovery goes down mid-session: `DISCOVERYRETRY` timer attempts to reconnect; existing connections in `.servers.SERVERS` still work +- The gateway blocks in a `while` loop at startup until at least one discovery connection is available (if `DISCOVERYREGISTER:1b`): `gateway.q:535-540` +- `autodiscovery` function on each process is called by discovery when it restarts, triggering `retrydiscovery[]` + +### Process Attributes + +Override `.proc.getattributes` to publish attributes to discovery: + +```q +.proc.getattributes:{[] + `date`tables!(rdbpartition;tables`.)} +``` + +The gateway uses these for intelligent routing (e.g., route to HDB that has the required date range). + +--- + +## 4. EOD Event Sequence + +Sources: `tickerplant.q`, `rdb.q`, `wdb.q`, `gateway.q` + +### Complete EOD Timeline + +``` +T=EOD time (from .eodtime.nextroll) + +TP: .z.ts fires → calls endofday[] + TP: calls .u.end[] → publishes final batch of data + TP: calls endofday on each subscriber (RDB and WDB): async `endofday[date]` + TP: increments date, opens new log file + +RDB: receives endofday[date] + RDB: .rdb.endofday[date;processdata] + if reloadenabled: + store row counts in eodtabcount + notify gateway of updated attributes (async) + return (WDB will call reload[] later) + else: + call .rdb.writedown[hdbdir; date] + sort each table (.sort.sorttab) + save to HDB: .Q.en[hdbdir; table] → .Q.par[hdbdir;date;tablename] + notify HDBs: send (`reload;date) to each + call .save.postreplay[hdbdir;date] + reset timeout + +WDB: receives endofday[date] (if in saveandsort or save mode) + WDB: .wdb.endofday[pt;processdata] + endofdaysave[savedir;pt] — flush remaining in-memory rows to temp partition + if saveandsort: endofdaysort[savedir;pt;...] OR endofdaymerge[...] for partbyattr modes + sort tables using .sort.sorttab (possibly with peach on .z.pd worker processes) + movetohdb[savedir;hdbdir;pt] — rename temp partition to HDB + .save.postreplay[hdbdir;pt] + if permitreload: doreload[pt] + informgateway(`reloadstart;`) — gateway blocks new multi-server queries + getprocs[pt] each reloadorder — send reload message to each HDB/RDB/IDB + wait for all callbacks or eodwaittime timeout + flushend[] + informgateway(`reloadend;`) — gateway re-enables queries, refreshes attributes + increment currentpartition::pt+1 + +HDB: receives (`reload;date) + HDB: load[hdbpath] — reloads entire HDB from disk + +Gateway: receives reloadstart + .gw.seteod[1b] + sends timeout errors to any in-flight multi-server queries + +Gateway: receives reloadend + .gw.seteod[0b] + refreshes server attributes + .gw.runnextquery[] +``` + +### Extension Pattern: Custom EOD Hooks + +**Option 1 — Post-writedown hook (runs after all tables saved, before HDB reload):** +```q +// In your proctype.q settings file or process code: +.save.postreplay:{[hdbdir;date] + .lg.o[`postreplay;"custom EOD work for ",string date]; + // your work here + } +``` + +**Option 2 — Table manipulation before save:** +```q +// Manipulate table before it is written +.save.savedownmanipulation:enlist[`trade]!enlist {[t] + // t is the table data; return modified version + select from t where size>0 + } +``` + +**Option 3 — Override `endofday` itself (advanced):** +```q +// Wrap the existing function +.rdb.endofday_orig:.rdb.endofday; +.rdb.endofday:{[date;processdata] + .lg.o[`eod;"custom pre-EOD logic"]; + // ... custom work ... + .rdb.endofday_orig[date;processdata]; + .lg.o[`eod;"custom post-EOD logic"]; + } +endofday:.rdb.endofday; // must re-assign root alias +``` + +### RDB Reload Mode (for WDB-managed RDB) + +When `.rdb.reloadenabled:1b` (typically when WDB is used): +1. RDB at EOD stores row counts per table in `eodtabcount` dict +2. RDB does NOT write to disk — returns immediately +3. WDB calls `reload[date]` on RDB after HDB is updated +4. `reload[]` drops the first `eodtabcount[t]` rows from each table (old data) +5. This keeps the RDB running without a full restart + +--- + +## 5. Config Layering — Common Mistakes + +**Mistake 1 — Not using guard pattern:** +```q +// WRONG — this will be overwritten by config files +.myns.myvar:100 + +// CORRECT — this preserves any value set earlier +.myns.myvar:@[value;`.myns.myvar;100] +``` + +**Mistake 2 — Setting config after startup:** +```q +// If you set a var in a process code file that loads AFTER config, +// you will overwrite what was in the config file. +// Config loads BEFORE process code. +// Use guard pattern in process code too. +``` + +**Mistake 3 — Wrong config file location:** +```q +// File must be at $KDBAPPCONFIG/settings/myproctype.q +// NOT at $KDBAPPCONFIG/myproctype.q +// NOT at $KDBCONFIG/myproctype.q (unless you want it to apply to all app deployments) +``` + +**Mistake 4 — Command-line override syntax:** +```bash +# Correct: full dot-path with leading dash +q torq.q ... -.servers.HOPENTIMEOUT 5000 -.rdb.reloadenabled 1 + +# Wrong: without leading dot +q torq.q ... -HOPENTIMEOUT 5000 +``` + +**Mistake 5 — parentproctype confusion:** +``` +parentproctype code/config loads BEFORE proctype code/config. +If sort process uses -parentproctype wdb, it gets: + config: settings/wdb.q then settings/sort.q + code: code/wdb/ then code/sort/ +``` +``` + +--- \ No newline at end of file diff --git a/.claude/skills/torq-developer/torq-patterns.md b/.claude/skills/torq-developer/torq-patterns.md new file mode 100644 index 000000000..ffec5b744 --- /dev/null +++ b/.claude/skills/torq-developer/torq-patterns.md @@ -0,0 +1,330 @@ +# TorQ Patterns Reference + +## Namespace Table + +| Namespace | Script | Purpose | +|---|---|---| +| `.proc` | `torq.q` | Process identity, startup hooks, loading utilities | +| `.lg` | `torq.q` | Logging: `.lg.o`/`.lg.e`/`.lg.w`, JSON log support, hook `.lg.ext` | +| `.timer` | `timer.q` | Multiple function timer: `.timer.repeat`/`.timer.once`/`.timer.timer` | +| `.hb` | `heartbeat.q` | Heartbeat publication and monitoring | +| `.sub` | `pubsub.q` | Subscription: `.sub.subscribe`, `.sub.SUBSCRIPTIONS`, `.sub.getsubscriptionhandles` | +| `.ps` | `pubsub.q` | Publish/subscribe: `.ps.publish`, `.ps.subscribe`, `.ps.initialise` | +| `.api` | `api.q` | Function documentation, search, memory usage | +| `.dotz` | `dotz.q` | Handler management: `.dotz.set`, `.dotz.getcommand` | +| `.servers` | `trackservers.q` | Connection registry: `.servers.SERVERS`, `.servers.getservers`, `.servers.startup` | +| `.clients` | `trackclients.q` | Inbound connection tracking: `.clients.clients` | +| `.access` | `controlaccess.q` | Access control: user whitelist, IP restriction, function restriction | +| `.usage` | `logusage.q` | Query logging: `.usage.usage`, LEVEL config | +| `.os` | `os.q` | OS utilities: `.os.sleep`, `.os.pth`, `.os.ren`, `.os.deldir` | +| `.gc` | `gc.q` | Garbage collection wrapper: `.gc.run[]` | +| `.loader` | `loader.q` | CSV/flat file loading utilities | +| `.cache` | `cache.q` | Function result caching: `.cache.add`, `.cache.maxsize` | +| `.async` | `async.q` | Async helpers: `.async.deferred`, `.async.postback` | +| `.cmp` | `compression.q` | Compression utilities | +| `.email` | `email.q` | Email sending via C lib | +| `.tz` | `timezone.q` | Timezone conversions: `.tz.lg`, `.tz.gl` | +| `.eodtime` | `eodtime.q` | EOD roll time: `.eodtime.nextroll`, `.eodtime.dailyadj` | +| `.ds` | `datasource.q` | Data source abstraction | +| `.rdb` | `rdb.q` | RDB-specific: `endofday`, `reload`, `subscribe`, `writedown` | +| `.wdb` | `wdb.q` | WDB-specific: `endofday`, `savetodisk`, modes, merge | +| `.gw` | `gateway.q` | Gateway: `asyncexec`, `syncexec`, `asyncexecjpt`, `servers`, `queryqueue` | +| `.pm` | `permissioning.q` | Permission management: `.pm.allowed` | +| `.ldap` | `ldap.q` | LDAP authentication | +| `.readonly` | `writeaccess.q` | Read-only mode | +| `.zpsignore` | `zpsignore.q` | Async message pattern filtering | +| `.mon` | `monitor.q` | Process monitoring | +| `.save` | `dbwriteutils.q` | EOD hooks: `.save.postreplay`, `.save.savedownmanipulation`, `.save.manipulate` | +| `.sort` | `sort.q` | Sort configuration from sort.csv: `.sort.sorttab`, `.sort.getsortcsv` | +| `.merge` | `merge.q` | WDB merge operations for partbyattr modes | +| `.finspace` | (FinSpace code) | AWS FinSpace integration flag: `.finspace.enabled` | + +--- + +## Config Layering + +``` +KDBCONFIG/settings/default.q ← lowest priority (TorQ defaults) +KDBCONFIG/settings/{parentproctype}.q +KDBCONFIG/settings/{proctype}.q +KDBCONFIG/settings/{procname}.q +KDBSERVCONFIG/settings/default.q +KDBSERVCONFIG/settings/{parentproctype}.q +KDBSERVCONFIG/settings/{proctype}.q +KDBSERVCONFIG/settings/{procname}.q +KDBAPPCONFIG/settings/default.q ← app-level (FSP etc) +KDBAPPCONFIG/settings/{parentproctype}.q +KDBAPPCONFIG/settings/{proctype}.q +KDBAPPCONFIG/settings/{procname}.q ← highest priority +command line: -.ns.var value ← overrides everything +``` + +--- + +## Logging Patterns + +```q +// Standard logging +.lg.o[`label;"info message"] // INF → out_ log file +.lg.e[`label;"error message"] // ERR → err_ log file (should be empty in production) +.lg.w[`label;"warning message"] // WRN → out_ log file + +// Extend all logging (fires on every .lg.* call) +.lg.ext:{[loglevel;procname;label;msg] + // e.g. push to monitoring system, alert on ERR + if[loglevel=`ERR; .email.send[...]] + } + +// JSON logs: start with -jsonlogs flag +// torq.q sets .lg.format to JSON formatter +``` + +--- + +## Handler Management + +```q +// Set a new handler (correct way) +.dotz.set[`.z.pc; mynewhandler] + +// Chain onto existing handler +.dotz.set[`.z.pc; + {.myns.mypc[y]; x@y} // call mine then existing + @[value; .dotz.getcommand[`.z.pc]; {;}] // get current (or no-op) + ] + +// Get current handler command (the function assigned, before TorQ wrapping) +.dotz.getcommand[`.z.pc] +``` + +Available handlers (from `handlers.md`): +- `logusage.q` — modifies pw, po, pg, ps, pc, ws, ph, pp, pi, exit, timer +- `controlaccess.q` — modifies pw, pg, ps, ws, ph, pp, pi +- `trackclients.q` — modifies po, pg, ps, ws, pc +- `trackservers.q` — modifies pc, timer +- `zpsignore.q` — modifies ps +- `writeaccess.q` — modifies pg, ps, ws, ph, pp +- `ldap.q` — modifies pw + +--- + +## Timer Scheduling Modes + +```q +// Repeating timer +// mode 0: reschedule at T0+P (fixed rate — can backlog if slow) +// mode 1: reschedule at T1+P (from when it fired — can drift) +// mode 2: reschedule at T2+P (from when it finished — safest for slow functions) +.timer.repeat[starttime; endtime; period; func; "description"; mode] +// mode defaults to 0 if omitted + +// One-shot timer +.timer.once[firetime; func; "description"] + +// Inspect timer +.timer.timer // table of scheduled functions (active=0b means disabled by error) + +// Timer is driven by .z.ts; must start kdb+ with -t or use system"t " +``` + +--- + +## Subscription Management + +```q +// Subscribe to tickerplant +// Returns dict: `subtables`tplogdate +subinfo:.sub.subscribe[ + `trade`quote; // tables (` = all) + `; // syms (` = all) + 1b; // retrieve schema? + 1b; // replay log? + handle // TP handle + ] + +// Get subscription handles (for a process type) +handles:.sub.getsubscriptionhandles[`tickerplant;();()!()] + +// Subscription state +.sub.SUBSCRIPTIONS // table of active subscriptions + +// Custom upd handler (must be at ROOT namespace) +upd:{[t;x] t insert x} // simplest form +``` + +--- + +## IPC Patterns + +### Synchronous +```q +h:first exec w from .servers.getservers[`proctype;`hdb;()!();1b;1b] +result:h (`.proc.procname;`) +result:h ({select from trade where date=.z.d};`) +``` + +### Asynchronous (Fire and Forget) +```q +neg[h] (`.myns.myfunc; arg1; arg2) +neg[h] (::) // flush to ensure message is sent +``` + +### Deferred Synchronous (client blocks) +```q +// Client sends async, blocks on handle until server responds via neg[.z.w] +neg[h] (`.gw.asyncexec; query; `rdb) +result:h[] // block +``` + +### Postback (client continues) +```q +// Server will call .myns.callback[result] on client +neg[h] (`.gw.asyncexecjpt; query; `rdb; raze; `.myns.callback; 0Wn) +``` + +### Broadcast to Multiple +```q +// Send same message to multiple handles +(neg each handles) @\: (`.myns.func; arg) +// Then flush each +(neg each handles) @\: (::) +``` + +### .async Helpers +```q +// Deferred sync to multiple handles (blocks until all respond) +.async.deferred[handles; (query; arg)] // returns (success_list; result_list) + +// Postback to multiple handles +.async.postback[handles; query; `callbackfunc] // returns success vector immediately +``` + +--- + +## Connection Management Patterns + +```q +// Declare connections at startup +.servers.CONNECTIONS:`rdb`hdb`gateway + +// Then call startup (usually done automatically by TorQ) +.servers.startup[] + +// Get all available HDB handles +hdbs:exec w from .servers.getservers[`proctype;`hdb;()!();1b;0b] + +// Get single handle with selection strategy +h:.servers.gethandlebytype[`hdb;`roundrobin] // rotate through HDBs +h:.servers.gethandlebytype[`hdb;`any] // random +h:.servers.gethandlebytype[`hdb;`last] // most recently used + +// Get handle to specific process name +h:first exec w from .servers.getservers[`procname;`rdb1;()!();1b;1b] + +// Attribute-based lookup +// Find HDB with specific date range +req:`date`tables!(enlist 2024.01.01; enlist enlist`trade) +tab:.servers.getservers[`proctype;`hdb;req;1b;0b] +// attribmatch column shows which attributes matched + +// Block until required processes are available +.servers.startupdepcycles[`tickerplant; 10; 100] // wait up to 100*10s=1000s +.servers.startupdependent[`hdb] // wait forever +``` + +--- + +## EOD Extension Patterns + +```q +// Post-writedown hook (runs after save, before HDB reload) +.save.postreplay:{[hdbdir;date] + .lg.o[`postreplay;"custom EOD work"]; + // copy files, send emails, trigger external systems + } + +// Table manipulation before save +.save.savedownmanipulation:enlist[`trade]!enlist {[data] + select from data where size > 0 + } + +// Custom HDB notification message +.rdb.hdbmessage:{[date] (`customreload; date)} + +// Register custom function at EOD completion +.proc.addinitlist {[] .lg.o[`init;"EOD handlers registered"]} +``` + +--- + +## Caching + +```q +// Cache result of expensive function +cachedresult:.cache.add[{expensivequery[]}; `mycachekey; `status] + +// Config +.cache.maxsize:500000000 // max total cache size in bytes +.cache.maxindividual:50000000 // max size of single cache entry +``` + +--- + +## Error Trapping Patterns + +```q +// Safe value retrieval +result:@[value;`myfunc;{.lg.e[`label;x];::}] + +// Safe IPC call +result:@[handle;(query;arg);{.lg.e[`ipc;x]; 0#()}] + +// Safe multi-arg call +result:.[func;(arg1;arg2);{.lg.e[`label;x]}] + +// Re-raise after logging +result:@[func;arg;{.lg.e[`label;x];'x}] // re-signal the error + +// Return null table on error +result:@[{select from trade where date=x};date;{0#trade}] +``` + +--- + +## Process Attribute Pattern + +Expose process capabilities for intelligent gateway routing: + +```q +// In your proctype settings or init code: +.proc.getattributes:{[] + `tables`date!(tables`.;.rdb.rdbpartition) + } + +// Gateway then routes by attribute: +// servertype=enlist[`tables]!enlist enlist`trade → only processes with trade table +// servertype=enlist[`date]!enlist enlist 2024.01.01 → only processes with that date +``` + +--- + +## .api Functions Quick Reference + +```q +.api.f `pattern // find all matching functions/vars (case insensitive) +.api.p `pattern // public only +.api.u `pattern // user-defined only +.api.s "*pattern*" // search function bodies +.api.m[] // memory usage table (sorted by size) +.api.mem[0b] // memory without evaluating views +.api.whereami[.z.s] // name of current function +.api.exportconfig[`.myns] // table of all config vars in namespace +.api.exportallconfig[] // all TorQ namespaces +.api.torqnamespaces // list of TorQ namespaces + +// Document a function +.api.add[`.ns.func; 1b; "description"; "params"; "return"] +``` +``` + +--- \ No newline at end of file diff --git a/.claude/skills/torq-developer/torq-process-templates.md b/.claude/skills/torq-developer/torq-process-templates.md new file mode 100644 index 000000000..645245ef9 --- /dev/null +++ b/.claude/skills/torq-developer/torq-process-templates.md @@ -0,0 +1,434 @@ +# TorQ Process Templates + +## process.csv Format (Complete) + +```csv +host,port,proctype,procname,U,localtime,g,T,w,load,startwithall,extras,qcmd +``` + +| Column | Required | Description | Example | +|---|---|---|---| +| `host` | Yes | Hostname or `localhost` | `localhost` | +| `port` | Yes | Port number; `{KDBBASEPORT}+N` expands via env vars | `{KDBBASEPORT}+2` | +| `proctype` | Yes | Process type (drives code/config loading) | `rdb` | +| `procname` | Yes | Unique process name | `rdb1` | +| `U` | No | Path to access list file (user:password per line) | `${TORQAPPHOME}/appconfig/passwords/accesslist.txt` | +| `localtime` | No | 1=local time for logs, 0=UTC | `1` | +| `g` | No | GC mode: 1=immediate (safer), 0=deferred (faster) | `1` | +| `T` | No | Query timeout in seconds (0=unlimited) | `180` | +| `w` | No | Max workspace in MB | `4000` | +| `load` | No | File or directory to load | `${KDBCODE}/processes/rdb.q` | +| `startwithall` | No | 1=included in `torq.sh start all` | `1` | +| `extras` | No | Additional command-line parameters | `-s -2 -parentproctype wdb` | +| `qcmd` | No | q executable name (default `q`) | `q` | + +### Finance Starter Pack process.csv (reference) + +```csv +host,port,proctype,procname,U,localtime,g,T,w,load,startwithall,extras,qcmd +localhost,{KDBBASEPORT}+1,discovery,discovery1,...,1,0,,,${KDBCODE}/processes/discovery.q,1,,q +localhost,{KDBBASEPORT},segmentedtickerplant,stp1,...,1,0,,,${KDBCODE}/processes/segmentedtickerplant.q,1,-schemafile ${TORQAPPHOME}/database.q -tplogdir ${KDBTPLOG},q +localhost,{KDBBASEPORT}+2,rdb,rdb1,...,1,1,180,,${KDBCODE}/processes/rdb.q,1,,q +localhost,{KDBBASEPORT}+3,hdb,hdb1,...,1,1,60,4000,${KDBHDB},1,,q +localhost,{KDBBASEPORT}+4,hdb,hdb2,...,1,1,60,4000,${KDBHDB},1,,q +localhost,{KDBBASEPORT}+5,wdb,wdb1,...,1,1,,,${KDBCODE}/processes/wdb.q,1,,q +localhost,{KDBBASEPORT}+6,sort,sort1,...,1,1,,,${KDBCODE}/processes/wdb.q,1,-s -2 -parentproctype wdb,q +localhost,{KDBBASEPORT}+7,gateway,gateway1,...,1,1,,4000,${KDBCODE}/processes/gateway.q,1,,q +localhost,{KDBBASEPORT}+8,kill,killtick,,1,0,,,${KDBCODE}/processes/kill.q,0,,q +localhost,{KDBBASEPORT}+9,monitor,monitor1,,1,0,,,${KDBCODE}/processes/monitor.q,0,,q +localhost,{KDBBASEPORT}+10,tickerlogreplay,tpreplay1,,1,0,,,${KDBCODE}/processes/tickerlogreplay.q,0,,q +localhost,{KDBBASEPORT}+11,housekeeping,housekeeping1,...,1,0,,,${KDBCODE}/processes/housekeeping.q,1,,q +localhost,{KDBBASEPORT}+12,reporter,reporter1,...,1,0,,,${KDBCODE}/processes/reporter.q,0,,q +localhost,{KDBBASEPORT}+14,feed,feed1,,1,0,,,${KDBAPPCODE}/tick/feed.q,1,,q +localhost,{KDBBASEPORT}+15,segmentedchainedtickerplant,sctp1,...,1,0,,,${KDBCODE}/processes/segmentedtickerplant.q,1,-parentproctype segmentedtickerplant,q +localhost,{KDBBASEPORT}+16,sortworker,sortworker1,,1,1,,,${KDBCODE}/processes/wdb.q,1,-parentproctype wdb,q +localhost,{KDBBASEPORT}+18,metrics,metrics1,...,1,1,,,${KDBAPPCODE}/processes/metrics.q,1,,q +``` + +Source: TorQ-Finance-Starter-Pack `appconfig/process.csv` + +--- + +## Minimal New Process + +```q +// myproc.q — load via: q torq.q -load code/processes/myproc.q -proctype myproc -procname myproc1 + +\d .myproc + +// Config with guard pattern (all overridable from config files or command line) +targetproctype:@[value;`targetproctype;`hdb] +pollinterval:@[value;`pollinterval;0D00:01] + +// Main logic +run:{[] + h:first exec w from .servers.getservers[`proctype;targetproctype;()!();1b;0b]; + if[null h; .lg.w[`run;"no handle to ",string targetproctype]; :()]; + res:@[h;myquery;{.lg.e[`run;"query failed: ",x]}]; + .lg.o[`run;"got ",string count res," rows"]; + } + +\d . + +// Extend CONNECTIONS to include our target +.servers.CONNECTIONS:distinct .servers.CONNECTIONS,`.myproc.targetproctype + +// Register timer +if[@[value;`.timer.enabled;0b]; + .timer.repeat[.proc.cp[];0Wp;.myproc.pollinterval;(`.myproc.run;`);"Poll target process"]]; + +// Document public function +.api.add[`.myproc.run;1b;"Poll target process and log result count";"[]";"()"]; +``` + +--- + +## Feedhandler Template + +```q +// feedhandler.q +// Start: q torq.q -load code/processes/feedhandler.q -proctype feed -procname feed1 + +\d .feed + +// Config (all overridable) +targettp:@[value;`targettp;`tickerplant] +publishinterval:@[value;`publishinterval;0D00:00:01] +tables:@[value;`tables;`trade`quote] + +// Track TP handle +tph:`int$() + +// Connect to tickerplant +gettph:{[] + tph::first exec w from .servers.getservers[`proctype;targettp;()!();1b;1b]; + if[null tph; .lg.w[`gettph;"no tickerplant available"]]; + tph} + +// Publish data to TP +publishtrade:{[] + if[null h:gettph[]; :()]; + data:(enlist .z.p; enlist `AAPL; enlist 150.5; enlist 100i; enlist 0b; enlist " "; enlist "N"; enlist `nasdaq); + @[neg[h]; (`upd;`trade;flip `time`sym`price`size`stop`cond`ex`src!data); + {.lg.e[`publish;"failed to publish: ",x]}] + } + +publishquote:{[] + if[null h:gettph[]; :()]; + data:(enlist .z.p; enlist `AAPL; enlist 150.4; enlist 150.6; enlist 100; enlist 100; enlist " "; enlist "N"; enlist `nasdaq); + @[neg[h]; (`upd;`quote;flip `time`sym`bid`ask`bsize`asize`mode`ex`src!data); + {.lg.e[`publish;"failed to publish: ",x]}] + } + +\d . + +// Set CONNECTIONS +.servers.CONNECTIONS:distinct .servers.CONNECTIONS,`.feed.targettp + +// Register timers +if[@[value;`.timer.enabled;0b]; + .timer.repeat[.proc.cp[];0Wp;.feed.publishinterval;(`.feed.publishtrade;`);"Publish trade"]; + .timer.repeat[.proc.cp[];0Wp;.feed.publishinterval;(`.feed.publishquote;`);"Publish quote"]]; +``` + +--- + +## Custom RDB Template + +```q +// Custom RDB — load via: q torq.q -load code/processes/rdb.q -proctype myrdb -procname myrdb1 +// Or set -parentproctype rdb to inherit standard RDB code and just override + +// Override specific settings BEFORE rdb.q is loaded +// (put this in appconfig/settings/myrdb.q) +\d .rdb +ignorelist:`heartbeat`logmsg`myinternaltable // add to ignore list +gc:1b // enable GC at EOD +reloadenabled:1b // use WDB-managed reload +subscribeto:`trade`quote // only these tables +\d . + +// Post-subscribe hook: runs after subscription is set up +// Put in code/myrdb/ directory or use .proc.addinitlist +.proc.addinitlist {[] + .lg.o[`init;"custom RDB initialised: subscribed to "," " sv string .rdb.subtables] + } + +// Custom upd hook (root namespace) +upd:{[t;x] + // Call default insert + t insert x; + // Custom: publish count to gateway + if[t=`trade; + if[count h:exec w from .servers.getservers[`proctype;`gateway;()!();0b;0b]; + neg[first h] (`.myapp.tradetick; count value t)] + ] + } + +// Custom EOD post-hook +.save.postreplay:{[hdbdir;date] + .lg.o[`eod;"custom post-EOD: ",string date]; + // e.g. update a control table, send email + } +``` + +--- + +## Custom WDB Template + +```q +// Settings for WDB (appconfig/settings/wdb.q or appconfig/settings/wdb1.q) +\d .wdb +mode:`saveandsort // saveandsort | save | sort +writedownmode:`default // default | partbyattr | partbyenum | partbyfirstchar +maxrows:enlist[`]!enlist 500000 // default max rows before writedown (per table) +// or per-table: +// maxrows:`trade`quote!200000 100000 +settimer:0D00:00:30 // check row counts every 30s +gc:1b // GC after each save +eodwaittime:0D00:00:30 // wait 30s for reload callbacks at EOD +reloadorder:`hdb`rdb // reload HDBs first, then RDBs +\d . + +// Sort configuration (sort.csv) +// Format: tablename,att,sortKey1,sortKey2,... +// trade,p,sym,time ← parted attribute on sym, sorted by sym then time +// quote,p,sym,time ← parted attribute on sym, sorted by sym then time +``` + +--- + +## Gateway Extension Template + +```q +// Gateway extension (put in appconfig/settings/gateway.q or code/gateway/) + +// Allow sync calls (disabled by default) +.gw.synccallsallowed:1b + +// Enable permissioning +.gw.permissioned:1b + +// Extend server connection hook: called when new processes register +.servers.addprocscustom:{[connectiontab;procs] + // Call default behaviour + .gw.runnextquery[]; + .gw.addserversfromconnectiontable[.servers.CONNECTIONS]; + // Custom: log new process registration + .lg.o[`addprocs;"new processes registered: "," " sv string exec procname from connectiontab]; + } + +// Extend on-connect hook: runs when gateway connects to a backend +.servers.connectcustom:{[f;connectiontab] + .gw.addserversfromconnectiontable[.servers.CONNECTIONS]; + f@connectiontab; + // Custom: refresh attributes + .lg.o[`connect;"connected to "," " sv string exec procname from connectiontab]; + }@[value;`.servers.connectcustom;{{[x]}}] +``` + +--- + +## setenv.sh Template (Full) + +```bash +#!/bin/bash +if [ "-bash" = "$0" ]; then + dirpath="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +else + dirpath="$(cd "$(dirname "$0")" && pwd)" +fi + +# TorQ framework location +export TORQHOME=/path/to/TorQ/latest +# Application location (can be same as TORQHOME for simple setups) +export TORQAPPHOME=/path/to/myapp + +export KDBCONFIG=${TORQHOME}/config +export KDBCODE=${TORQHOME}/code +export KDBLOGS=${TORQAPPHOME}/logs +export KDBHTML=${TORQHOME}/html +export KDBLIB=${TORQHOME}/lib +export KDBHDB=${TORQAPPHOME}/hdb +export KDBWDB=${TORQAPPHOME}/wdbhdb +export KDBTPLOG=${TORQAPPHOME}/tplogs +export KDBAPPCONFIG=${TORQAPPHOME}/appconfig +export KDBAPPCODE=${TORQAPPHOME}/code +export KDBBASEPORT=6000 + +# Process CSV location +export TORQPROCESSES=${KDBAPPCONFIG}/process.csv + +# Optional: separate DQC/DQE databases +export KDBDQCDB=${TORQAPPHOME}/dqe/dqcdb/database +export KDBDQEDB=${TORQAPPHOME}/dqe/dqedb/database + +# q executable path (if not in PATH) +export QCMD=q +export RLWRAP=rlwrap +export QCON=qcon +``` + +--- + +## TorQ Project Structure (Full) + +``` +myapp/ +├── appconfig/ +│ ├── process.csv ← Process definitions +│ ├── passwords/ +│ │ ├── accesslist.txt ← user:password entries +│ │ ├── default.txt ← default outbound connection password +│ │ └── rdb.txt ← rdb-specific outbound password +│ └── settings/ +│ ├── default.q ← App-wide config overrides +│ ├── rdb.q ← RDB-specific config +│ ├── hdb.q ← HDB-specific config +│ ├── gateway.q ← Gateway-specific config +│ └── rdb1.q ← Process-name-specific config +├── code/ +│ ├── common/ ← Loaded by all processes +│ │ └── myutilities.q +│ ├── rdb/ ← Loaded by rdb proctype +│ │ └── customrdb.q +│ ├── hdb/ ← Loaded by hdb proctype +│ │ └── customhdb.q +│ ├── gateway/ ← Loaded by gateway proctype +│ │ └── customgw.q +│ └── processes/ ← Process entry-point files +│ └── myfeed.q +├── hdb/ ← HDB data directory ($KDBHDB) +├── wdbhdb/ ← WDB temp storage ($KDBWDB) +├── tplogs/ ← TP log files ($KDBTPLOG) +├── logs/ ← TorQ log files ($KDBLOGS) +├── database.q ← Table schema definitions +└── setenv.sh ← Environment setup script +``` + +--- + +## TorQ Schema File (database.q) + +```q +// FSP example (database.q) +quote:([]time:`timestamp$(); sym:`g#`symbol$(); bid:`float$(); ask:`float$(); + bsize:`long$(); asize:`long$(); mode:`char$(); ex:`char$(); src:`symbol$()) +trade:([]time:`timestamp$(); sym:`g#`symbol$(); price:`float$(); size:`int$(); + stop:`boolean$(); cond:`char$(); ex:`char$(); side:`symbol$()) +``` + +Rules: +- `time` must be first column (`timestamp` type for TP compatibility) +- `sym` must be second column with `` `g# `` attribute +- Passed to TP via `-schemafile database.q` in process.csv extras +- TP validates that all incoming `upd` messages have time+sym as first two columns + +--- + +## sort.csv Format + +```csv +tablename,att,sortKey1,sortKey2,... +trade,p,sym,time +quote,p,sym,time +``` + +| Column | Description | +|---|---| +| `tablename` | Table to configure | +| `att` | Attribute to apply to sort key 1: `p`=parted, `g`=grouped, `u`=unique, `s`=sorted | +| `sortKey1...N` | Columns to sort by (in order) | + +--- + +## Quick Deployment Checklist + +- [ ] `setenv.sh` defines all required env vars +- [ ] `process.csv` has correct ports (no conflicts), `startwithall=1` for production processes +- [ ] `appconfig/passwords/accesslist.txt` created with `admin:admin` (or secure equivalent) +- [ ] Schema file (`database.q`) has `time` first, `sym` second, `` `g# `` on sym +- [ ] `sort.csv` configured for all tables +- [ ] `hdb/`, `wdbhdb/`, `tplogs/`, `logs/` directories created +- [ ] All processes in CONNECTIONS list (RDB needs `tickerplant`; gateway needs `rdb`, `hdb`) +- [ ] `.proc.getattributes` overridden on HDB/RDB to expose date/table attributes for gateway routing +- [ ] Timer enabled (`-t 1000` in extras or `system"t 1000"` in code) where required (RDB requires it) +- [ ] EOD hooks (`.save.postreplay`, `.save.savedownmanipulation`) tested against replay +- [ ] Log directory writable by process user +``` + +--- + +## Environment Variables + +| Variable | Description | +|---|---| +| `KDBCONFIG` | Base configuration directory | +| `KDBCODE` | TorQ code directory | +| `KDBLOGS` | Log file directory | +| `KDBHTML` | HTML files for web interfaces | +| `KDBLIB` | Supporting library files | +| `KDBAPPCONFIG` | Application config directory (overrides KDBCONFIG) | +| `KDBAPPCODE` | Application code directory | +| `KDBBASEPORT` | Base port (FSP default: 6000, processes at KDBBASEPORT+offset) | +| `KDBHDB` | HDB directory | +| `KDBTPLOG` | TP log directory | +| `KDBWDB` | WDB temp storage directory | + +--- + +## Deployment Directory Layout + +``` +deploy/ +├── bin/ +│ ├── torq.sh # Process management script (Linux only) +│ └── setenv.sh # Sets all env vars; source before torq.sh +├── TorQ/latest/ # TorQ framework (never modify) +│ ├── torq.q +│ ├── code/ +│ └── config/ +└── TorQApp/latest/ # Application layer + ├── appconfig/ + │ ├── process.csv + │ ├── settings/ # App-specific config overrides + │ └── passwords/ # Connection password files + └── code/ # App-specific code +``` + +--- + +## Config and Code Layering (complete order) + +For each of `KDBCONFIG`, `KDBSERVCONFIG`, `KDBAPPCONFIG`: +1. `settings/default.q` +2. `settings/{parentproctype}.q` +3. `settings/{proctype}.q` +4. `settings/{procname}.q` + +For code loading, for each of `KDBCODE`, `KDBSERVCODE`, `KDBAPPCODE`: +1. `common/` directory +2. `{parentproctype}/` directory +3. `{proctype}/` directory +4. `{procname}/` directory +5. `handlers/` directories + +Then: `-load` file(s) specified on command line. + +--- + +## torq.sh Commands + +```bash +./deploy/bin/torq.sh start all # start all startwithall=1 processes +./deploy/bin/torq.sh stop all # graceful stop all +./deploy/bin/torq.sh debug rdb1 # start rdb1 in foreground (-debug -nopi) +./deploy/bin/torq.sh qcon rdb1 admin:admin # qcon into running process +./deploy/bin/torq.sh summary # show status table +./deploy/bin/torq.sh procs # list all processes +./deploy/bin/torq.sh print rdb1 # print startup command line +./deploy/bin/torq.sh top rdb1 # show top.q stats +./deploy/bin/torq.sh stop rdb1 -force # kill -9 +``` + +--- \ No newline at end of file diff --git a/docs/claude-skill.md b/docs/claude-skill.md new file mode 100644 index 000000000..aa945d7af --- /dev/null +++ b/docs/claude-skill.md @@ -0,0 +1,39 @@ +# Claude Skill for TorQ + +A [Claude Code skill](https://docs.claude.com/en/docs/claude-code/skills) ships with this repo under [.claude/skills/torq-developer/](../.claude/skills/torq-developer/). It teaches Claude the TorQ conventions it wouldn't otherwise know — the guard pattern for config, `.servers.*` for connections, `.timer.repeat` for scheduling, `.api.add` for public functions, the EOD lifecycle, and the two-stage workflow for adding a new process. + +With the skill loaded, Claude produces code that fits a TorQ codebase instead of plausible-looking q that ignores the framework. + +## What's in the skill + +- `SKILL.md` — the core rules (namespace, config, logging, handlers, timers, schemas, subscriptions, connections, gateway patterns, q-language pitfalls, EOD). Always loaded. +- `torq-internals.md` — startup order, EOD sequence, gateway request lifecycle, discovery protocol. +- `torq-patterns.md` — namespace table, IPC and subscription patterns, caching, async helpers, error-trapping idioms. +- `torq-process-templates.md` — `process.csv` columns, `setenv.sh`, `torq.sh` commands, deployment checklist, and templates for minimal process, feedhandler, RDB, WDB, gateway. +- `q-language-reference.md` — general q/kdb+ reference (not TorQ-specific). +- `kdb-ecosystem.md` — integrating kdb+ with Python (PyKX/embedPy), Grafana, REST/HTTP, WebSockets, C API. + +The companion files are loaded on demand when the current task matches their topic. + +## Using the skill when deploying TorQ + +The skill lives in `.claude/skills/torq-developer/` so that Claude Code auto-discovers it whenever it is invoked inside a clone of this repo. No setup required — open the repo and ask Claude to make a change. + +When you build your own application on top of TorQ (e.g. a fork, a starter-pack-style layered repo, or a project that vendors TorQ as a submodule), you have three options for making the skill available: + +1. **Per-project** — copy `.claude/skills/torq-developer/` into your downstream repo at the same path. The skill travels with the repo and every teammate who clones it gets it automatically. + +2. **Per-user (global)** — copy the directory to `~/.claude/skills/torq-developer/` on your machine. The skill is then available to Claude Code in every project you open, not just TorQ ones. Useful if you work across several TorQ-based repos. + + ```bash + mkdir -p ~/.claude/skills + cp -r /path/to/TorQ/.claude/skills/torq-developer ~/.claude/skills/ + ``` + +3. **Submodule / symlink** — if you vendor TorQ as a git submodule, symlink `.claude/skills/torq-developer` in your parent repo to the path inside the submodule. The skill stays pinned to the TorQ version you're running. + +## Extending the skill for your deployment + +The shipped skill covers the framework. Any conventions specific to your deployment — naming, allowed ports, required process attributes, site-local EOD hooks — belong in a separate skill or in `CLAUDE.md` at the root of your downstream repo. Do not bloat `SKILL.md`: a skill that tries to encode every edge case drowns out the rules that matter most. + +The blog post [Can Claude Talk TorQ?](https://www.dataintellect.com/thoughts/) walks through designing a skill and using it to build a new TorQ process end-to-end. \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 2786fbbef..0e799972e 100755 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -14,6 +14,7 @@ nav: - Monitoring: monit.md - Visualisation: visualisation.md - Testing: unittesting.md + - Claude Skill: claude-skill.md - TorQ Blog Posts: blog.md