Add lite presets for starting and evaluating minimal task suite

Author: Moshe Kimhi

README.md

@@ -200,6 +200,14 @@ python -m src.start_task -a
 This will launch five task_workers each for `dbbench-std` and `os-std` tasks and automatically connect them
 to the controller on port 5000. **After executing this command, please allow approximately 1 minute for the task setup to complete.** If the terminal shows ".... 200 OK", you can open another terminal and follow step 4.
 
+#### Lite preset (laptops / limited RAM)
+
+If you want to start with minimal concurrency (1 worker per task), use the lite preset:
+
+```bash
+python -m src.start_task -a --config configs/start_task_lite.yaml
+```
+
 ### Step 4. Start the assigner
 
 This step is to actually start the tasks.
@@ -210,6 +218,12 @@ If everything is correctly configured so far, you can now initiate the task test
 python -m src.assigner
 ```
 
+If you started the task server with the lite preset, you can also run the lite evaluation preset:
+
+```bash
+python -m src.assigner --config configs/assignments/lite.yaml
+```
+
 ## Next Steps
 
 If you wish to launch more tasks or use other models, you can refer to the content

configs/assignments/lite.yaml

@@ -0,0 +1,22 @@
+# Lite preset: evaluate only low-resource tasks with minimal concurrency.
+#
+# Usage:
+#   python -m src.assigner --config configs/assignments/lite.yaml
+
+import: definition.yaml
+
+concurrency:
+  task:
+    dbbench-std: 1
+    os-std: 1
+  agent:
+    gpt-3.5-turbo-0613: 1
+
+assignments:
+  - agent:
+      - gpt-3.5-turbo-0613
+    task:
+      - dbbench-std
+      - os-std
+
+output: "outputs/{TIMESTAMP}"

configs/start_task_lite.yaml

@@ -0,0 +1,15 @@
+# Lite preset: start only low-resource tasks.
+# Intended for laptops / limited RAM.
+#
+# Usage:
+#   python -m src.start_task -a --config configs/start_task_lite.yaml
+#
+# You can still override tasks at runtime:
+#   python -m src.start_task -a --config configs/start_task_lite.yaml -s dbbench-std 2 os-std 2
+
+definition:
+  import: tasks/task_assembly.yaml
+
+start:
+  dbbench-std: 1
+  os-std: 1