Next-Best-Question

A research codebase for adaptive question selection ("next-best question") and efficient reciprocal matching experiments. The benchmark supports two application scenarios: romantic matchmaking and job interview.

Environment Setup

1. Python Environment

• Python: 3.10+ recommended

• Create virtual environment and install dependencies:

  python -m venv .venv
  source .venv/bin/activate  # On Windows: .venv\Scripts\activate
  pip install -U pip
  pip install -r requirements.txt

• Upgrade pip and install the sync package in editable mode:

  python3 -m pip install --upgrade pip setuptools wheel
  pip install -e .

2. API Keys Configuration

This project requires API keys for both OpenAI and Google Gemini models. Set up your environment variables:

OpenAI API Key

# For OpenAI models (gpt-4o, gpt-4.1, gpt-5-mini, etc.)
export OPENAI_API_KEY="your_openai_api_key_here"

Google Gemini API Key

# For Google Gemini models (gemini-2.5-flash, etc.)
export GOOGLE_API_KEY="your_google_api_key_here"

Verify API Keys

To verify your API keys are properly set:

echo $OPENAI_API_KEY
echo $GOOGLE_API_KEY

Note: Both API keys are required as the experiments use different models for different components (analysis, question generation, evaluation).

Supported Scenarios

Scenario	Code	Scripts	Description
Romantic Matchmaking	mm	interviews/data/scripts/mm/	Understanding respondent self-information and mate preferences across 10 core dimensions (emotional, cultural, social, etc.)
Job Interview	job	interviews/data/scripts/job/	Understanding interviewee qualifications across 10 core dimensions (work identity, hard skills, track record, etc.)

Each scenario uses scenario-specific prompts for answer analysis, data synthesis, and evaluation, ensuring that the question generation and profiling pipeline is tailored to the domain.

Running the Main Experiments

The codebase includes three main experiments. Experiments A and B apply to both scenarios; Experiment C is scenario-independent.

Warning: Full runs may incur nontrivial financial costs due to LLM API usage. Consider starting with smaller sample sets for testing.

Experiment A: Main Performance Comparison (Profiling)

This experiment runs the core next-best question selection algorithm on synthetic scenarios.

A1. Romantic Matchmaking Scenario

Location: interviews/exp/shs/run_mm_0927_ours.sh

What it does:

• Runs 5 iterations of conversation simulation on 50 synthetic matchmaking personas (mm_0927_01 to mm_0927_50)
• Evaluates the "ours" method for adaptive question selection at rounds 3, 4, and 5

How to run:

cd ./interviews
bash exp/shs/run_mm_0927_ours.sh

Baselines (same scenario):

bash exp/shs/run_mm_0927_rand.sh    # Random question selection baseline
bash exp/shs/run_mm_0927_llms.sh    # LLM-only question generation baseline

A2. Job Interview Scenario

Location: interviews/exp/shs/run_job_ours.sh

What it does:

• Runs 5 iterations on 9 synthetic job interview personas (3 positions × 3 detail levels)
• Evaluates the "ours" method at rounds 3, 4, and 5
• Positions: data scientist, product manager, sales manager

How to run:

cd ./interviews
bash exp/shs/run_job_ours.sh

Baselines (same scenario):

bash exp/shs/run_job_rand.sh    # Random question selection baseline
bash exp/shs/run_job_llms.sh    # LLM-only question generation baseline

Note: All scripts contain --a_api_key XXX which can be removed if using OpenAI/Gemini models with environment variables set.

Expected output: Results will be saved in interviews/exp/results/ directory.

Experiment B: Reciprocal Matching Evaluation

This experiment evaluates the quality of matches found by the question selection algorithm. Applicable to the matchmaking scenario only.

Location: interviews/exp/shs/run_reciprocal_match.sh

What it does:

• Compares "ours" method against "trivial" baseline
• Evaluates matching quality using average scoring
• Tests top-k matching performance (k=1,2,5,10)
• Uses results from Experiment A1

Prerequisites: Must run Experiment A1 first. Additionally, you need results from two separate question-set interviews:

1. Self-information interviews using data/var/split_subtopics/q_all_eval_1002_general_100_tri.json
2. Mate-preference interviews using data/var/split_subtopics/q_all_eval_p_1002_general_100_tri.json for all users with "_pref" suffix

How to run:

cd ./interviews
bash exp/shs/run_reciprocal_match.sh

Expected output: Matching evaluation results in interviews/exp/results/reps/ directory.

Experiment C: Efficient Matching on RandUser-100K

This experiment tests the efficiency of matching algorithms on large-scale synthetic data (scenario-independent).

Location: eval/efficient_matching/run.sh

What it does:

• Tests FAISS-based efficient matching against brute-force methods
• Evaluates different index types: IVF1024,Flat, IVF1024,SQ8, HNSW32
• Tests various similarity thresholds (0.6, 0.65, 0.7, 0.75)
• Uses 100K synthetic user vectors

How to run (from root directory):

bash eval/efficient_matching/run.sh

Expected output:

• CSV results: eval/efficient_matching/sweep_results_uniform.csv
• JSON results: eval/efficient_matching/sweep_results_uniform.json
• Plot: eval/efficient_matching/recall_uniform.png

Data Synthesis Pipeline

The synthetic benchmark data (scripts, questionnaires, profiles) is generated using the pipeline in eval/conversation/. This pipeline supports both matchmaking and job interview scenarios.

Generating New Synthetic Instances

Single-instance generation:

cd eval/conversation

# Matchmaking scenario (default)
python construct.py 5 --prefix mm --topic_num 10 --gender male

# Job interview scenario
python construct.py 5 --prefix jb --topic_num 10 --scenario job

Batch generation (multiple instances per batch, more efficient):

# Matchmaking scenario
python construct_batch.py 2 --batch_size 10 --prefix mm --scenario mm

# Job interview scenario with sequential insight generation
python construct_batch.py 2 --batch_size 10 --prefix jb --scenario job --seq_insight

Sequential generation (for better intra-persona consistency):

python construct.py 5 --prefix jb --scenario job --seq_concrete
python construct_batch.py 2 --batch_size 10 --prefix jb --scenario job --seq_insight --seq_concrete

Generating Preference Data (Matchmaking Only)

python construct_pref.py 5 --prefix mm

Experiment Workflow

Recommended Execution Order

1. Run Experiment A1 (Matchmaking) — generates core matchmaking results (main result)
2. Run Experiment A2 (Job Interview) — generates job interview results
3. Run Experiment B (Reciprocal Matching) — depends on A1 results
4. Run Experiment C (Efficient Matching) — can run independently

Running All Experiments

# Terminal 1: Matchmaking experiments (from interviews directory)
cd ./interviews
bash exp/shs/run_mm_0927_ours.sh
bash exp/shs/run_mm_0927_rand.sh

# Terminal 2: Job interview experiments (from interviews directory)
cd ./interviews
bash exp/shs/run_job_ours.sh
bash exp/shs/run_job_rand.sh

# After matchmaking experiments complete:
bash exp/shs/run_reciprocal_match.sh

# Terminal 3: Efficient matching (from root directory)
bash eval/efficient_matching/run.sh

Understanding the Results

Experiment A Results

• Location: interviews/exp/results/
• Content: Conversation logs, question selection decisions, and performance metrics
• Key metrics: ACC@T, AR@T (accuracy and answer recall at round T)

Experiment B Results

• Location: interviews/exp/results/reps/
• Content: Matching quality comparisons between methods
• Key metrics: Hit@K

Experiment C Results

• Location: eval/efficient_matching/
• Content: Efficiency benchmarks and recall curves
• Key metrics: Recall, Latency

Project Structure

Next-Best-Question/
├── interviews/                  # Main benchmark framework
│   ├── main.py                  # Entry point for conversation experiments
│   ├── parse.py                 # CLI argument parsing
│   ├── config/ours/             # Agent A configuration (scenario-specific paths)
│   ├── data/
│   │   ├── scripts/mm/          # Matchmaking personas (200 files)
│   │   ├── scripts/job/         # Job interview personas (9 files)
│   │   ├── questionnaires/      # Ground-truth evaluation questions
│   │   └── reps/                # Pairing files for reciprocal matching
│   ├── exp/shs/                 # Experiment shell scripts
│   └── src/                     # Interview agents and evaluation
│       ├── agent_a.py           # Default LLM agent
│       ├── agent_a_ours.py      # Our method (next-best question selection)
│       ├── agent_b.py           # Respondent simulation agent
│       ├── agent_c.py           # Evaluator agent
│       ├── prompts.py           # Scenario-specific prompts for agents A/B/C
│       └── eval_matcher/        # Reciprocal matching evaluation
├── src/sync/                    # Core algorithm package
│   ├── agent/                   # Analysis, relevance, question selection agents
│   ├── llm/                     # LLM interface, prompts, pricing
│   ├── qgen/                    # Question generation pipeline
│   ├── workflow/                # Conversation workflow orchestration
│   └── tool/                    # Matching tools
├── eval/
│   ├── conversation/            # Data synthesis pipeline (both scenarios)
│   └── efficient_matching/      # FAISS matching experiments
└── data/var/                    # Pre-computed question analysis and embeddings
    ├── split_subtopics/         # Question pools (mm + job)
    └── rel_cache/               # Analysis caches and embeddings (mm + job)

Troubleshooting

Common Issues

1. API Key Errors: Ensure both OPENAI_API_KEY and GOOGLE_API_KEY are set
2. Permission Denied: Run scripts with bash script_name.sh instead of ./script_name.sh
3. Missing Dependencies: Reinstall requirements with pip install -r requirements.txt
4. Out of Memory: Reduce --num-pairs in Experiment C for smaller datasets

Model Configuration

Default models are configured in src/sync/configs.py. You can modify these if needed:

• AA_A_MODEL: Analysis model (default: gpt-4.1)
• DEFAULT_T_MODEL: Transformation model (default: GPT-4o)
• EMBED_MODEL: Embedding model (default: OpenAI text-embedding-3-large)

Additional Resources

• Configuration files: interviews/config/ contains experiment-specific settings
• Data directory: interviews/data/ contains synthetic profiles and questionnaires
• Scenario prompts: interviews/src/prompts.py for interview agents; src/sync/llm/prompts.py for analysis agents; eval/conversation/prompts.py for data synthesis

For detailed parameter descriptions and advanced usage, see the individual script documentation in their respective directories.