Skip to content

Kappa on subcorpus: file selection in coder comparisons + n/a for non-computable stats - #15

Draft
qwertygc wants to merge 5 commits into
masterfrom
vibe/kappa-subcorpus-b8791c
Draft

qwertygc wants to merge 5 commits into
masterfrom
vibe/kappa-subcorpus-b8791c

Conversation

@qwertygc

Copy link
Copy Markdown
Owner

Summary

  • Add a "Select files" button to the Coder Comparisons dialog so users can restrict the Cohen's Kappa / agreement statistics to a chosen subcorpus of text files, instead of always using the entire corpus
  • By default all text files are used (behavior unchanged); the selected subcorpus is displayed in the dialog and included in the text report header
  • Add unit tests (tests/test_report_coder_comparisons.py) covering full corpus vs subcorpus calculation, Kappa change under filtering, perfect-agreement Kappa, and unknown file ids

Details

Coders may not have coded every interview, so calculating Kappa over the whole corpus can be misleading. This PR lets the user pick the relevant files (multi-select with checkboxes, using the existing DialogSelectItems component) before pressing Run. The filtering is applied in calculate_agreement_for_code (src/qualcoder/reports.py), leaving the statistics logic itself unchanged.

Verification

  • python -m py_compile on the two modified modules and the new test file
  • Logic verified against an in-memory SQLite project with two files/two coders: full corpus gives characters=20, kappa=0.2987; subcorpus of file 1 gives characters=10, kappa=1.0, agree_coded_only=100.0; filtering ignores unknown file ids
  • New unit tests run headless (QT_QPA_PLATFORM=offscreen) with a minimal sqlite schema, same pattern as tests/test_qualcoder.py

mistral-vibe and others added 2 commits October 12, 2026 16:29
Allow users to select a subset of text files before running the coder
comparison statistics, so Cohen's Kappa can be calculated on a subcorpus
instead of the entire corpus. All text files are used by default.

- reports.py: new select_files dialog, selected_file_ids filter in
  calculate_agreement_for_code, file names shown in dialog and report
- GUI/ui_report_comparisons.py: Select files button and files label
- tests/test_report_coder_comparisons.py: unit tests for subcorpus
  filtering of the Kappa and agreement statistics

Co-authored-by: qwertygc <qwertygc@users.noreply.github.com>
When a code is not applied by either coder within the selected files,
the Kappa and percentage statistics cannot be computed. Previously the
internal sentinel strings 'zerodiv'/'zero div' were displayed in the
tree, which is frequent now that a subcorpus can be selected. Display
'n/a' instead and add a unit test for this case.

Co-authored-by: qwertygc <qwertygc@users.noreply.github.com>
@qwertygc qwertygc changed the title Allow Kappa calculation on a selected subcorpus in coder comparisons Kappa on subcorpus: file selection in coder comparisons + n/a for non-computable stats Oct 12, 2026
mistral-vibe and others added 3 commits October 13, 2026 06:13
… pyuic6

The dialog previously had no .ui source file in src/GUI_UIs, so the
generated .py file had been edited directly. Add the Qt Designer
template matching the current dialog (including the Select files
button and files label) and regenerate the .py file with pyuic6 so
future .ui edits will not overwrite hand-made changes.

Co-authored-by: qwertygc <qwertygc@users.noreply.github.com>
Replace the 'Select files' text button with a 32x32 icon button
(mdi file-multiple, consistent with the other icon buttons in the
dialog) and show the selection state in a label next to it:
'Files: All' when no subcorpus filter is active, otherwise
'Files: N' with the number of selected files. The text report
header lists the selected file names for traceability.

Co-authored-by: qwertygc <qwertygc@users.noreply.github.com>
…utton

Co-authored-by: qwertygc <qwertygc@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants