Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ All notable changes to Fair Code are documented here, newest first.
- `scripts/check_traction.py --live` (#1148) compares stars, forks, watchers and contributors with GitHub, and a weekly `traction-live` workflow keeps one issue up to date.
- `scripts/external-links-ignore.txt` (#1174): URLs the link check does not request, each with a reason, counted as "ignored" in the report.
- `scripts/generate_og_images.py --force` (#1144).
- `assets/og/card-inputs.json` and `assets/og-light/card-inputs.json` (#1197): a hash of each OG card's kicker, title, subtitle and theme. `scripts/generate_og_images.py` re-renders a card whose hash changed, so editing an explainer's title or summary rewrites only its two PNGs, and `scripts/check_generated_files_current.py` fails when a stored hash is stale.
- Six more explainers, each run against a real audit: [AUC Gaps by Group](explainers/auc-gaps-by-group.md), [Decision Curve Analysis by Group](explainers/decision-curve-analysis-by-group.md) and [Cost-Sensitive Evaluation](explainers/cost-sensitive-evaluation.md) (#1137), then [Popularity Bias](explainers/popularity-bias.md), [Two-Sided Fairness in a Marketplace](explainers/two-sided-fairness.md) and [Position Bias and Click Models](explainers/position-bias-and-click-models.md) (#1134). Explainer count `108 → 114`.
- `explainers/code/<slug>.py` (#1125): the first Python block of an explainer that names its own path is written to that file by `scripts/build_explainers.py`, so the "run from the repo root" sentence is true for 52 explainers, and the files are checked as generated files and compiled by a test.
- `scripts/check_external_links.py` and a weekly workflow (#1127): reports dead external links in the docs and explainers as one issue that is opened, updated and closed by the run. A report, never a gate.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -396,7 +396,7 @@ Fair-Code/
├── assets/ # web assets (CSS + JS)
│ ├── fonts/
│ ├── icons/ # favicons + PWA icons
│ ├── og/ # dark-theme OG share images, per explainer
│ ├── og/ # dark-theme OG share images, per explainer, + card-inputs.json hashes
│ ├── og-light/ # light-theme counterparts
│ ├── profiler-engine.js, profiler-ui.js, profiler-compare.js, profiler.css # client-side profiler
│ ├── benchmark-dashboard.js, benchmark.css # client-side benchmark results dashboard
Expand Down
119 changes: 119 additions & 0 deletions assets/og-light/card-inputs.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
{
"accuracy-equality": "49a8e53abcfd36cd",
"accuracy-not-enough-healthcare-ai": "cfa71ec6aee06b3c",
"adversarial-debiasing": "97e2dcb0a1cf7139",
"adverse-action-notices": "c20f171ae19c42c6",
"aggregation-bias": "4756adb217bc5d21",
"ai-hallucinations": "465e6464ec8b343e",
"ai-objectivity-myth": "ff1e319c9a7c5ad7",
"algorithmic-impact-assessment": "0a9e6dd473362376",
"algorithmic-recourse": "aa4de1fbf95f4cf4",
"auc-gaps-by-group": "953f4ee5a4486ff4",
"automation-bias": "5830cbecda031763",
"base-rate-fallacy": "0eb85b4c7f9facd6",
"benchmark": "f1f3eefa0122ed27",
"bias-variance-tradeoff": "869c2485ed86d6af",
"bootstrap-confidence-intervals": "c45eb156a5a93bf5",
"calibration": "eb59e8f8e4a15b75",
"causal-graphs-for-fairness": "0a443a1add267042",
"choosing-a-fairness-metric": "6fe3ba8e4fa759c7",
"choosing-a-mitigation-strategy": "46589f3639e4026d",
"class-imbalance": "220940d0b66ad07a",
"clinical-score-miscalibration": "7ffaf7fa4079ce81",
"compounding-disparities": "a53e87e453984195",
"conditional-demographic-parity": "aa6568c66f76c64c",
"confidence-intervals-for-ratios": "bcb920f7913d5035",
"confounding-variable": "8f363cfa329731fe",
"confusion-matrix": "551b92e3a45dd10f",
"cost-sensitive-evaluation": "8fc69c30c461b3f2",
"counterfactual-explanation": "31b6994d5bc31a49",
"counterfactual-fairness": "bf3925e82950b82c",
"cross-validation-for-fairness": "298f55c93b703ea5",
"data-leakage": "d2a813f57410e64c",
"datasheets-for-datasets": "35d022e8d212b1ce",
"decision-curve-analysis-by-group": "3c2c5b65945732f7",
"delayed-feedback": "d9387c52b5617c61",
"demographic-parity": "bbbc1ce7cd92b670",
"differential-privacy": "2ce738c36e78a9ed",
"differential-validity": "cc57641892235065",
"disaggregated-evaluation": "79f4e1a1a7efbea0",
"discharge-leakage-in-readmission-models": "c9dc46cc7c700aeb",
"disparate-impact": "b26db72fa2418486",
"disparate-treatment": "6929cf6c9b468786",
"distribution-shift": "ce6a2daa70dffada",
"distributionally-robust-optimization": "107c95f668534f61",
"equal-opportunity": "74768f93f0db0f86",
"equalized-odds": "653a367b414b79c9",
"explanation-disagreement": "478de67841fd73ef",
"fair-representation-learning": "7ad9e3d08e0c4c8b",
"fairness-accuracy-pareto-frontier": "84137bd6f16d5e9d",
"fairness-accuracy-tradeoff": "69690159249ee3da",
"fairness-in-ranking": "f4ec117017864a38",
"fairness-metric-conflicts": "2c4e25c30ac054d4",
"fairness-regularization": "284e36fd09d4cf78",
"fairness-through-unawareness": "96667161a48a25cc",
"fairness-without-demographics": "5901de145317fed4",
"false-positives-vs-false-negatives": "d49d31fd10f2e35b",
"feature-importance-by-group": "5eb7d17a693a6399",
"feedback-loop-bias": "6df7b0537bb821ae",
"geographic-proxies-and-redlining": "c9c2d46347226fed",
"group-specific-thresholds": "dab77a7b178e8e1a",
"hiring-pipeline-adverse-impact": "e2d8307d4aca8c39",
"home": "5815b73a981a226c",
"how-ai-detects-patterns": "ca68fe25b3040c26",
"individual-fairness": "439ebf1ca29472f4",
"inferring-protected-attributes": "39007b3ed5e63a28",
"intersectional-bias": "eb755ce3beeb8d6c",
"label-bias": "b42f1fb95a39e102",
"label-shift-and-fairness": "3ac950f884555d6e",
"leave-one-group-out-evaluation": "a3d4e0b3a73675bc",
"lime": "c12594b60f8cfec5",
"maxmin-fairness": "2183ea5f0b0afffe",
"measurement-bias": "dc3ec3789eb13667",
"medical-imaging-representation-gaps": "d93bab5462c76ee6",
"minimum-detectable-gap": "d772c7cc6384b439",
"missing-data-bias-ehr": "c751b42a298791a4",
"mitigation-strategies": "4b96f1553d5b2ba9",
"ml-bias": "ea29326712a964e7",
"model-cards": "dd04ea3a48866c77",
"model-drift": "e2073bb56c783e1b",
"monitoring-fairness-in-production": "d1ce48ff72f08248",
"multiple-comparisons": "51614d38c24b6532",
"neural-networks": "7d68b451ef07758e",
"obermeyer-cost-proxy": "c9513d88896cfc74",
"partial-dependence-by-group": "9fdd1310aca7bf7c",
"path-specific-effects": "b49f2e631079a990",
"permutation-test-fairness-gaps": "e3d1e9f914d6f948",
"popularity-bias": "32c62401af506226",
"position-bias-and-click-models": "0fbc1492340cb43a",
"precision-recall-curve": "7cefc31ffc2947f5",
"predictive-parity": "2efce5a691dbf655",
"profiler": "f2bd04bdc7b34402",
"protected-attribute": "f341474d7458dd80",
"proxy-entanglement": "b02eeb0f89a992bd",
"proxy-variables": "d4e096592b777661",
"race-correction-clinical-algorithms": "7f634b1edbe40dc4",
"race-in-clinical-models": "2a6edc43839d0e7e",
"randomized-post-processing": "746432984d9139f7",
"reinforcement-learning": "c24d351eabe2468e",
"reject-inference": "653326cc4416629f",
"reject-option-classification": "7dce2ab136ea355e",
"reweighing": "79c4b9538006155c",
"risk-based-pricing-gaps": "290a11b20a26502c",
"roc-curve-auc": "a8102fe2492dce1e",
"sampling-bias": "2d272eafcf22f947",
"seed-and-split-sensitivity": "b6424478f6bd1abe",
"selection-bias": "0193acb60fa501d4",
"shap-values": "3e931af0faa692ad",
"simpsons-paradox": "1defd2a2b1f01db0",
"small-group-shrinkage": "8dc852af3a8e2b63",
"stratified-test-sets-for-fairness": "a87909ddd86261c0",
"subgroup-fairness": "e8fcd5f326164c50",
"supervised-learning": "9fe887ef812fb76e",
"survivorship-bias": "a8f2e63032d44132",
"treatment-equality": "a4c5743aa97338ce",
"two-sided-fairness": "a967c8dc931803db",
"underdiagnosis-bias": "803b7a36bff91d7f",
"unsupervised-learning": "ed76876a5cb0bf20",
"utilization-as-a-label": "ebe034a70fa4a28a"
}
119 changes: 119 additions & 0 deletions assets/og/card-inputs.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
{
"accuracy-equality": "e073914c149b311f",
"accuracy-not-enough-healthcare-ai": "92e4fa54474ae59c",
"adversarial-debiasing": "3918ecf465a25141",
"adverse-action-notices": "d2587a84c93829ff",
"aggregation-bias": "f11cb25022e42c1f",
"ai-hallucinations": "8d83b4fed8c5b1a5",
"ai-objectivity-myth": "9717f41444ed08b4",
"algorithmic-impact-assessment": "8549a9728d545f3e",
"algorithmic-recourse": "3b210653a5f29280",
"auc-gaps-by-group": "45047499e0ad8349",
"automation-bias": "ae54c290a972cacb",
"base-rate-fallacy": "147093dcc10e14cb",
"benchmark": "161dade406717667",
"bias-variance-tradeoff": "3eb72e87f3398e63",
"bootstrap-confidence-intervals": "ba649d55621670f5",
"calibration": "20f4614e5ba9219c",
"causal-graphs-for-fairness": "3f963d93a87eccf1",
"choosing-a-fairness-metric": "b3c878ae9dbfee4e",
"choosing-a-mitigation-strategy": "79b65accace8c3d0",
"class-imbalance": "755ee7a95777f5d6",
"clinical-score-miscalibration": "57efd3b3e38508a1",
"compounding-disparities": "84e02c3261735761",
"conditional-demographic-parity": "669457462c043a42",
"confidence-intervals-for-ratios": "99dcdb2a9d0210df",
"confounding-variable": "b8fe77374ca57082",
"confusion-matrix": "21a48525bbcd464d",
"cost-sensitive-evaluation": "7e2afb03b32628f6",
"counterfactual-explanation": "adcecd6bcb638b5d",
"counterfactual-fairness": "18966a0eafc42a75",
"cross-validation-for-fairness": "25f934b1e2615205",
"data-leakage": "dc2c2662a3a9e66c",
"datasheets-for-datasets": "7be0b4506853d471",
"decision-curve-analysis-by-group": "a4d69eeb9faa6a99",
"delayed-feedback": "c1b227ed9592af68",
"demographic-parity": "f9e5ee188789b063",
"differential-privacy": "ba996b1948d46be4",
"differential-validity": "458065ba57ff023f",
"disaggregated-evaluation": "dbbb1d9718f847d0",
"discharge-leakage-in-readmission-models": "6611325f7e7c96f5",
"disparate-impact": "fcab9076ac22cc53",
"disparate-treatment": "02ba4382b01e599b",
"distribution-shift": "a2b141101006181a",
"distributionally-robust-optimization": "2fcb08945599b840",
"equal-opportunity": "6e0d673b2c6366e3",
"equalized-odds": "ef799c419c97ea8e",
"explanation-disagreement": "79437d3493f44819",
"fair-representation-learning": "e75713fa9138223a",
"fairness-accuracy-pareto-frontier": "15121b60e0ac11da",
"fairness-accuracy-tradeoff": "147de9d6ba714dc1",
"fairness-in-ranking": "a229ab2d039ce981",
"fairness-metric-conflicts": "e30e1755ed9fb525",
"fairness-regularization": "364833e26777bf78",
"fairness-through-unawareness": "7665613b9d628bf3",
"fairness-without-demographics": "8a2fd0dd20fe8eb7",
"false-positives-vs-false-negatives": "798b1aa6edf00f7c",
"feature-importance-by-group": "caa0a59908e3ebe8",
"feedback-loop-bias": "803650eee51b317b",
"geographic-proxies-and-redlining": "d56ee5e248c92f8f",
"group-specific-thresholds": "34f3386e2e56f447",
"hiring-pipeline-adverse-impact": "7069b1c40e86c33d",
"home": "55da9eb4489fc837",
"how-ai-detects-patterns": "f28dd187cce2d79b",
"individual-fairness": "c6a3ffb4bd57c1f1",
"inferring-protected-attributes": "700bb66b64f14591",
"intersectional-bias": "c094f2354a16d380",
"label-bias": "f4f032215afdfabd",
"label-shift-and-fairness": "5a539f55acadfb81",
"leave-one-group-out-evaluation": "26b203a1074733ba",
"lime": "f06799bf15bbc35a",
"maxmin-fairness": "9b2c1288b6dbccae",
"measurement-bias": "218aa7db6e5e7b7e",
"medical-imaging-representation-gaps": "950142212c087a6c",
"minimum-detectable-gap": "e0ef36f8c635f775",
"missing-data-bias-ehr": "da0061a1766b41cc",
"mitigation-strategies": "0266f9c02a3a5786",
"ml-bias": "f333fd4aa51d4f02",
"model-cards": "a4905f764148d649",
"model-drift": "ba907271971a848b",
"monitoring-fairness-in-production": "4a90656dc2c9eb21",
"multiple-comparisons": "2fc0c1dd59357b69",
"neural-networks": "fd86c44ee3be99b7",
"obermeyer-cost-proxy": "6e346c90f1b86c91",
"partial-dependence-by-group": "36d64f1e5fd6212a",
"path-specific-effects": "e7d31b1733d65035",
"permutation-test-fairness-gaps": "5d30f36df71f9a2f",
"popularity-bias": "99f4ca10bcd165c4",
"position-bias-and-click-models": "68b226c5792ad745",
"precision-recall-curve": "6adc0e6d9252fbca",
"predictive-parity": "fa2dff9235208c51",
"profiler": "9a762bbdc490fafb",
"protected-attribute": "4e971d2d36e5ac7c",
"proxy-entanglement": "80db811c5142ec48",
"proxy-variables": "5035908f6aa3bb69",
"race-correction-clinical-algorithms": "98cf9611df61a9d3",
"race-in-clinical-models": "f2ca6366609530eb",
"randomized-post-processing": "5e6521a75ea5a0cd",
"reinforcement-learning": "dff341178a7f36ed",
"reject-inference": "506afd1d6908fbd8",
"reject-option-classification": "18ddcf1e51a08763",
"reweighing": "902a9ef1729b61f1",
"risk-based-pricing-gaps": "032d179b9a983326",
"roc-curve-auc": "ea1ebd6206320a76",
"sampling-bias": "5a8d7ca856a4c0f6",
"seed-and-split-sensitivity": "62a3349da6e04074",
"selection-bias": "08ec4a60361b3f98",
"shap-values": "028cee758649aac4",
"simpsons-paradox": "00d1f9410f6f18cf",
"small-group-shrinkage": "b05fa1a43a136443",
"stratified-test-sets-for-fairness": "85f95e5a000e162b",
"subgroup-fairness": "84ffe63349535bc5",
"supervised-learning": "9debb47a16c22c1e",
"survivorship-bias": "54e6041e7c51211d",
"treatment-equality": "81773ec50ebbbb5b",
"two-sided-fairness": "f606cb2306dc669d",
"underdiagnosis-bias": "920cb76a189747b9",
"unsupervised-learning": "cedc8fd3a5e1fdd4",
"utilization-as-a-label": "6059a37c236704bc"
}
9 changes: 8 additions & 1 deletion scripts/check_generated_files_current.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,12 @@
current dark and light OG image exists for every explainer, is non-empty,
and has the right dimensions; `tests/test_generate_images.py` already
verifies the *generator* satisfies that in a temp directory, so this
script checks the same thing for what's actually committed.
script checks the same thing for what's actually committed. Whether a
card's text is current is checked through the card-inputs.json file in
each OG folder (#1197): the generator stores a hash of each card's inputs
there, so it is compared byte-for-byte like the other text files, and an
explainer whose title or summary changed without a re-render reads as
stale.

This script is self-sufficient: it builds a fresh copy of the site itself
(via `git worktree`, with every tracked file and every untracked,
Expand Down Expand Up @@ -90,6 +95,8 @@
"faircode/_results_frozen/*.csv",
"faircode/_results_current/*.csv",
"explainers/code/*.py",
"assets/og/card-inputs.json",
"assets/og-light/card-inputs.json",
]

_JSONLD_DATE_LINE = re.compile(
Expand Down
58 changes: 49 additions & 9 deletions scripts/generate_og_images.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,18 @@
A card whose PNG already exists is skipped (#1144): the PNG bytes depend on the
Pillow and zlib build, so on another machine every committed image would come out
byte-different and a one-explainer change would touch hundreds of binary files.
Adding an explainer therefore writes only its two new PNGs. Pass --force to
re-render every card, which is what to do after editing a card's design in this
file or an explainer's title or summary (a changed title does not invalidate its
existing PNG).
Adding an explainer therefore writes only its two new PNGs.

Each theme folder also holds card-inputs.json (#1197), a hash of every card's
inputs (kicker, title, subtitle, theme). A card whose stored hash no longer matches,
because an explainer's title or summary changed, is re-rendered even though its PNG
exists, so editing one explainer rewrites only its two PNGs.
check_generated_files_current.py compares these files with a fresh regeneration, so
a stale hash fails CI. Pass --force to re-render every card, which is what to do
after editing a card's design in this file.
"""
import argparse
import hashlib
import json
import textwrap
from pathlib import Path
Expand All @@ -38,6 +44,8 @@
FONTS_DIR = ROOT / "assets" / "fonts"
DATA_JSON = ROOT / "assets" / "explainers-data.json"

HASHES_FILENAME = "card-inputs.json" # one per theme folder, see the module docstring

W, H = 1200, 630
MARK_GREEN = (79, 122, 91) # #4F7A5B - the corrected-F crossbar accent, fixed across themes, see logo.svg

Expand Down Expand Up @@ -180,13 +188,41 @@ def render_card(out_path, kicker, title, subtitle, theme, footer="thefaircode.xy
img.save(out_path, "PNG", compress_level=9)


def render_card_both_themes(slug, kicker, title, subtitle, force=True):
def card_inputs_hash(kicker, title, subtitle, theme):
"""A short, stable hash of everything a card's text depends on."""
payload = json.dumps([kicker, title, subtitle, theme], ensure_ascii=False)
return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:16]


def load_card_hashes():
"""Returns {theme: {slug: hash}} from each theme folder's card-inputs.json ({} if absent)."""
hashes = {}
for theme, colors in THEMES.items():
path = colors["out_dir"] / HASHES_FILENAME
hashes[theme] = json.loads(path.read_text(encoding="utf-8")) if path.is_file() else {}
return hashes


def write_card_hashes(hashes):
for theme, colors in THEMES.items():
path = colors["out_dir"] / HASHES_FILENAME
path.write_text(json.dumps(hashes[theme], indent=2, sort_keys=True) + "\n", encoding="utf-8")


def render_card_both_themes(slug, kicker, title, subtitle, force=True, stored=None, current=None):
"""Render the card in each theme; with force=False a theme whose PNG exists is left alone.
Returns how many PNGs were written."""

With `stored` ({theme: {slug: hash}}, from load_card_hashes) an existing PNG is only left
alone while its stored hash matches the card's inputs. The card's current hash is recorded
in `current`, when given. Returns how many PNGs were written."""
written = 0
for theme, colors in THEMES.items():
out_path = colors["out_dir"] / f"{slug}.png"
if out_path.exists() and not force:
digest = card_inputs_hash(kicker, title, subtitle, theme)
if current is not None:
current.setdefault(theme, {})[slug] = digest
up_to_date = stored is None or stored.get(theme, {}).get(slug) == digest
if out_path.exists() and up_to_date and not force:
continue
render_card(out_path, kicker, title, subtitle, theme)
written += 1
Expand All @@ -206,10 +242,12 @@ def main(force=False):
for colors in THEMES.values():
colors["out_dir"].mkdir(parents=True, exist_ok=True)
written = 0
stored = load_card_hashes()
current = {theme: {} for theme in THEMES}

def render(*args, **kwargs):
nonlocal written
written += render_card_both_themes(*args, force=force, **kwargs)
written += render_card_both_themes(*args, force=force, stored=stored, current=current, **kwargs)

render(
"home",
Expand Down Expand Up @@ -242,9 +280,11 @@ def render(*args, **kwargs):
subtitle=entry["summary"],
)

write_card_hashes(current)

total = (3 + len(entries)) * len(THEMES)
dirs = ", ".join(str(c["out_dir"].relative_to(ROOT)) for c in THEMES.values())
print(f"Wrote {written} of {total} OG images ({total - written} already existed; --force re-renders them) in {dirs}")
print(f"Wrote {written} of {total} OG images ({total - written} already current; --force re-renders them) in {dirs}")


if __name__ == "__main__":
Expand Down
Loading
Loading