Credit Card Fraud Detection

Given a card transaction, decide whether to let it through or flag it for a human to review — and justify that decision in money.

492 of 284,807 transactions are fraudulent. That's 0.17%, and it's what makes the problem hard: a model that flags nothing at all is right 99.87% of the time. Accuracy, the metric everyone reaches for first, is useless here.

So this isn't really about finding a better classifier. It's about choosing metrics and a cut-off that survive that imbalance, and being able to say what the result is worth. The answer here is $12,168 recovered over a 48-hour window.

Why it's genuinely difficult

Every mistake costs something different, and the two kinds trade against each other:

• Miss a fraud and the money is gone — the full transaction amount.
• Flag a legitimate transaction and an analyst wastes time reviewing it, and a customer may have a card declined for nothing.

Flag more and you catch more fraud but drown your review team. Flag less and fraud slips through. There's no setting that avoids both, so the job is picking the trade deliberately with the costs written down — which is what this does.

Everything below is produced by analysis.py.

pip install -r requirements.txt
python analysis.py

The data

284,807 card transactions over 48 hours. V1–V28 are PCA components — the original features were anonymised before release — leaving only Time and Amount directly interpretable.

1,081 exact duplicates (19 fraud, 1,062 legitimate) are removed first. A duplicated fraud landing on both sides of a split is a model scoring its own memory. After de-duplicating: 283,726 rows, 473 frauds (0.167%).

Time is seconds since the first transaction — an index, useless on future data. It's converted to hour of day, which is real signal, and dropped.

	Median	Mean
Fraud	9.82	123.87
Legitimate	22.00	88.41

Fraud has the lower median and the higher mean: most fraud is small — testing whether a stolen card works — and a few are very large. Total fraud in the dataset: 58,591.39.

Trap 1 — accuracy is unusable

"This is never fraud"
Accuracy	99.8743%
Frauds caught	0 of 107
Money recovered	0.00

Any accuracy figure quoted on this dataset is meaningless. Everything below uses precision, recall, and average precision instead.

Trap 2 — the split has to respect time

Transactions arrive in order, so a random split trains on Tuesday to predict Monday — something no deployed system can do.

Split	Average precision
Random	0.691
Time-ordered	0.776

Note which way that moved. The correct method scored higher, not lower. Sound methodology isn't the same as a pessimistic number, and choosing a split because it flatters the result would have been wrong in either direction.

Train: 198,608 transactions (366 frauds). Test: 85,118 (107 frauds).

Trap 3 — ROC-AUC picks the wrong model

Model	Average precision	ROC-AUC
Logistic regression	0.776	0.980
Random forest	0.812	0.951
Gradient boosting	0.772	0.944

The two metrics disagree about which model to ship. ROC-AUC ranks logistic regression best; average precision ranks random forest best.

ROC-AUC counts true negatives, and with 99.8% of the data legitimate that term swamps everything else — a model can rank the vast legitimate majority beautifully and still be poor at the 0.17% that matters. Average precision ignores true negatives entirely.

ROC-AUC isn't just optimistic here. It's actively misleading.

Trap 4 — average precision picks the wrong model too

Having established that average precision is the metric to trust, it also disagrees with the thing we actually care about. Scoring every candidate at its own cost-optimal threshold:

Model	Avg precision	Cost	Flagged	Recall	Precision
random_forest_100	0.812	2,396	167	82%	53%
random_forest_100_d12	0.817 ← best	2,694	138	79%	62%
random_forest_50_d10	0.815	2,795	524	87%	18%
logistic_regression	0.776	2,843	195	82%	45%
hist_gradient_boosting	0.772	2,931	561	85%	16%

Following average precision costs 298 more — 12%. Two precision-recall curves can enclose almost the same area and behave very differently at the one point you actually operate at.

random_forest_50_d10 is the sharpest illustration: average precision 0.815, essentially tied for best, but it flags 524 transactions to catch 107 frauds. No analyst would accept that queue, and no summary metric told you.

The lesson isn't that average precision is bad — it's far better than ROC-AUC here. It's that a summary statistic is a proxy, and when you have the real objective, optimise the real objective.

The part that matters: choosing a threshold

A probability is not a decision. Turning one into a decision needs a cost for each kind of mistake:

• Missed fraud costs the transaction amount — the money is gone.
• False alarm costs 3.00 of analyst time to review.

Both are assumptions, and both are arguable. That's the point of writing them down instead of leaving 0.5 as an unexamined default.

Threshold	Total cost	Recall
Do nothing	14,564.99	0%
Default 0.50	4,692.95	68.2%
Cheapest — 0.05	2,396.38	82.2%

Moving the threshold from 0.50 to 0.05 saves 2,296.57 — more than switching between any two models on the list. The default threshold is a worse decision than a worse model.

Net saving against doing nothing: 12,168.61 on a 48-hour test window.

The model as deployed

Random forest, threshold 0.05:

Class	Precision	Recall	F1	n
legitimate	1.000	0.999	0.999	85,011
fraud	0.527	0.822	0.642	107

            legitimate  fraud
legitimate       84932     79
fraud               19     88

	Count	Value
Frauds caught	88	12,405.61
Frauds missed	19	2,159.38
False alarms	79	237.00 to review

Roughly half of everything flagged is genuine fraud — against a base rate of 0.126% in the test window, that's a 419× improvement on random. It catches 82.2% of fraud by count and 85.2% by value, because the misses skew small.

An analyst reviews 167 transactions per 48 hours. That's a workload a person can actually carry, which is the constraint the cost model is really encoding.

Serving it

The trained model is committed (1.0 MB), so the API runs without the 150 MB dataset. Training needs the data; scoring doesn't.

pip install -r requirements-api.txt
uvicorn api:app

$ curl -X POST localhost:8000/predict \
    -H 'Content-Type: application/json' \
    -d '{"v": [...28 PCA components...], "amount": 149.62, "time": 3600}'

{"fraud_probability":0.91,"flagged":true,"threshold":0.05,"action":"review"}

POST	/predict	Score one transaction. ?threshold= overrides the default
POST	/predict/batch	Up to 1,000 at once
GET	/model	The model card — what's served, when it was trained, every candidate considered and why this one won
GET	/health	Liveness

The default threshold is 0.05, not 0.5. That number is derived from the cost model and shipped in metadata.json, but it's exposed as a parameter because the right cutoff belongs to whoever bears the cost.

Verified against held-out transactions the model never saw: 5 of 5 known frauds caught (probabilities 0.58–0.91), 0 false alarms on 5 legitimate ones.

docker build -t fraud-api . && docker run -p 8000:8000 fraud-api, or deploy with the included render.yaml. The service is stateless, so the free tier's lack of a disk costs nothing here.

What this shows

1. Accuracy is unusable. 99.87% is what detecting nothing scores.
2. ROC-AUC is misleading, not merely optimistic — it chose a different model than average precision, because true negatives swamp it.
3. Average precision picks the wrong model too, once there's a cost model to compare against — by 12%. A summary statistic is a proxy; when you have the real objective, optimise that.
4. A random split on time-ordered data trains on the future. Fixing it raised the score here, which is worth noticing: correct isn't a synonym for lower.
5. The threshold is a business decision. Moving it off the default saved more money than any change of model.

Each of those would have produced a confident, defensible, wrong answer.

The headline number of this project isn't a score. It's 12,168.61 recovered on a 48-hour window, under cost assumptions written down where you can argue with them.

The dataset

Not committed — creditcard.csv is 150 MB, over GitHub's file limit. Download it and drop it in data/:

Credit Card Fraud Detection — Machine Learning Group, ULB. The script exits with these instructions if the file is missing.

Files

analysis.py	The whole analysis
data/creditcard.csv	Not committed — see above
figures/	Generated plots

License

MIT — see LICENSE.