SA-693/two step classify

[dstewartons]

📌 Pull Request Template

Please complete all sections

✨ Summary

Adds two-step SOC LLM methods for Survey Assist classify, mirroring SIC in sic-classification-utils: unambiguous_soc_code (step 1) and formulate_open_question (step 2 when not codable). Introduces SOC_PROMPT_UNAMBIGUOUS and SOC_PROMPT_OPENFOLLOWUP. Keeps sa_rag_soc_code for demos and other callers; classify in survey-assist-api must use the two-step methods only.

Companion PR: survey-assist-api SA-693 — connects classify to these methods and removes SA-673 clear-winner logic from the API.

Release: version bumped to 0.1.4 — tag and publish after merge so downstream repos can pin the Git tag (replacing local path dependencies).

📜 Changes Introduced

• Feature implementation (feat:) / bug fix (fix:) / refactoring (chore:) / documentation (docs:) / testing (test:)

• Updates to tests and/or documentation

• Terraform changes (if applicable)

• llm/llm.py: unambiguous_soc_code, formulate_open_question (SIC-shaped chains, logging, correlation_id); sa_rag_soc_code aligned with sa_rag_sic_code (signature, parse fallbacks, prompt candidate metadata).

• llm/prompt.py: SOC_PROMPT_UNAMBIGUOUS, SOC_PROMPT_OPENFOLLOWUP; SA_SOC_PROMPT_RAG wording aligned with SIC (no SA-673 clear-winner instructions).

• models/response_model.py: UnambiguousResponse / OpenFollowUp docs aligned with SIC two-step naming.

• utils/constants.py: SOC prompt name constants for config consumers.

• tests/test_llm.py: mocked tests for unambiguous_soc_code and formulate_open_question (typed responses, call dict, job title normalisation).

• demos/llm/: example script and shortlist JSON (moved from src/.../llm_embedding_example.py).

• README.md: documents two-step vs single-shot RAG.

• pyproject.toml: version 0.1.4.

✅ Checklist

Please confirm you've completed these checks before requesting a review.

• Code is formatted using Black
• Imports are sorted using isort
• Code passes linting with Ruff, Pylint, and Mypy
• Security checks pass using Bandit
• API and Unit tests are written and pass using pytest
• Terraform files (if applicable) follow best practices and have been validated (terraform fmt & terraform validate)
• DocStrings follow Google-style and are added as per Pylint recommendations
• Documentation has been updated if needed

🔍 How to Test

1. Unit tests (no GCP required)

cd soc-classification-utils
poetry install
make all-tests
make check-python-nofix

*Verified 36 passed; make check-python-nofix clean.

Mocked tests in tests/test_llm.py cover unambiguous_soc_code and formulate_open_question (typed responses and call dicts).

2. Live LLM demo (this repo only)

What this proves: unambiguous_soc_code calls Gemini and returns valid JSON (step 1 only). Uses a fake shortlist from the JSON file — not the API or vector store.

Requires Vertex / Gemini credentials. No vector store or survey-assist-api.

cd soc-classification-utils
poetry install
poetry run python demos/llm/llm_embedding_example.py

The script uses a fixed shortlist in demos/llm/data/school_embed_short_list_soc.json and prints three JSON blocks (legacy get_soc_code, sa_rag_soc_code, then unambiguous_soc_code).

Check the third block has codable, class_code, alt_candidates, and reasoning. With the bundled school-teacher inputs and a shortlist of primary / secondary / special-needs codes, the model often returns codable: false and class_code: null, with 2313 as the leading entry in alt_candidates(ambiguous — step 2 would beformulate_open_questionin classify). Exact values can vary with the LLM; do not expect a fixedclass_code` every run.

Verified third block codable: false, class_code: null, top candidate 2313 at likelihood 0.9.

Full POST /classify checks (vector store + API, farm hand / manager curls) belong in the survey-assist-api companion PR — not repeated here.

Notes

• After merge, publish Git tag v0.1.4 so survey-assist-api and soc-classification-vector-store can pin it instead of path = "../soc-classification-utils".
• Classify consumers must call unambiguous_soc_code then formulate_open_questionwhen not codable — notsa_rag_soc_code`.