A credit default scoring project, step by step: the reproducible repository behind the UCI Default of Credit Card Clients data

The project and the repository

The task is to estimate, for each of 30,000 card holders, the probability of defaulting next month, so that a credit analyst can decide; the repository makes that estimate rebuildable by anyone.

The data are the UCI Default of Credit Card Clients table: 30,000 clients of a Taiwanese bank in 2005, with their credit limit, demographic fields, six months of repayment status, bill and payment amounts, and a binary target, default the following month. The raw file is never committed: a script downloads it and compares its SHA-256 fingerprint with the one stored in the repository, so a silent change of the source stops the pipeline.

The split is done once, stratified, 60% training, 20% validation, 20% test, with a fixed seed, and the fingerprint of each fold is stored too. Two models come with the repository: the base rate, which predicts the same probability for everyone, and a logistic regression inside a scikit-learn Pipeline, calibrated on the validation fold. The heavier model is left to the teams.

The rule is that nothing regenerable is versioned: no raw data, no folds, no trained model. The .gitignore refuses them and the commands rebuild them. The commit history follows the course, from the skeleton and the framing through data, split and models up to the tests, the notebooks, the milestone templates and the interface, so reading it is reading the project. Each team receives the whole thing as a pull request in its own repository: one member reads the changes, another merges, and everyone then runs the two commands.

Back to contents.

Two commands: make reproduce and make test

If the same command does not give the same numbers twice, then the data, the code, the seed or the environment has moved, and you need to know which.

make reproduce   # download and fingerprint, split, train, metrics
make test        # thirteen property tests, a few seconds
make compare     # milestone 2 only, reads the test fold once
make app         # one client, one decision

make reproduce chains four targets. The data step stops if the fingerprint changed, because data are not modified without an entry in the decision log. The split step writes the folds and their fingerprints, which a test then freezes. The training step fits the two provided models. The metrics step measures on the validation fold only and writes a metrics.json. The test fold is not read here at all: it is opened once, by make compare, at the second milestone.

The base rate is 22.12% in each fold, by construction of the stratified split. Its log-loss is not arbitrary: predicting the base rate $p$ for everyone gives exactly the entropy of the target, which is the floor any useful model must go below.

make test runs thirteen properties in a few seconds. A machine learning test is not "the AUC is good": it is something that must hold whatever happens. The schema contract (columns, row count, bounds, a 0/1 target, missing values); a split whose fingerprints never change and where no client appears in two folds; a model that reaches an AUC of 1 on separable data, otherwise the training code is broken; predictions unchanged by permuting rows; a reloaded model whose predictions match the trained one up to $10^{-12}$ in absolute terms; a naive model equal to the base rate; a logistic that beats it. Plus six robustness properties, described below.

Back to contents.

The guide notebook: the threshold is a decision, leaks are the enemy

The notebook imports the repository's code instead of copying it, and it shows that the threshold comes from costs, not from the model.

The guide notebook has seven sections, one per session: framing, the frozen split, the naive model then the logistic, the threshold, then the three milestones. Its first cell imports src/credit. A notebook that pastes its own copy of the code soon presents something other than what runs. Each section ends with a short list of things not to do, which together form a catalogue of data leakage: computing correlations on the full dataset before splitting, re-splitting with another seed, standardising before the split, tuning hyperparameters on the validation fold, peeking at the test fold.

Two guard-rails are visible in the logistic step. An assertion checks that the SEX column is not among the features; it is a principle, not a performance choice. And standardisation sits inside the Pipeline, so its means and variances are learned on the training fold only, which removes the most common leak of the course.

With calibrated probabilities, the threshold that minimises the expected cost of a refuse/accept decision depends only on the two error costs:

On the validation fold, the empirical minimum of the cost curve falls at 0.045, far from 0.1176, and the curve is almost flat between the two. That is the honest diagnosis: the baseline sorts low risks poorly. The repository keeps the theoretical threshold, which can be defended, writes it in config.yaml and justifies it in the decision log. A threshold edited by hand in a batch script or in the interface would be a decision nobody took.

Back to contents.

Why a model? Against business rules, in euros

Before any machine learning, a bank already refuses clients who are late on payments, so the model has to beat that rule in money, not only in AUC.

With these costs, the calibrated logistic does no better than the bluntest policy imaginable, since turning down nearly every applicant costs about as much. With 22% of defaults and a default 7.5 times more expensive than a lost client, an AUC of 0.72 does not separate low risks well enough. This is a fact about these costs and this model, not a verdict on machine learning, and it is the starting point of the second milestone.

Worked example: when both errors cost the same

1. Gap per file. $133 - 112 = 21$ euros in favour of the logistic, that is $21 / 133 \approx 16\%$ less than the rule.
2. Per year. On 6,000 files, about $21 \times 6000 = 126\,000$ euros; the deck, working from unrounded costs, quotes about 130,000 euros.
3. With its uncertainty. The validation AUC of 0.723 has a 95% bootstrap interval running from 0.706 up to 0.740, so a rival at 0.735 would not be established as better.

The lesson for a client memo: costs are assumptions given by the business, and the value of a model moves with them. The argument has to be shown across cost levels, with its interval.

Back to contents.

Six robustness tests, written before looking

A robust model is one for which you have written down, in advance, what it must do when the data move, and turned each statement into an automatic test.

• Interval. The bootstrap AUC is clearly away from 0.5 and its interval is narrower than 0.06.
• Noise. A 1% perturbation of the amounts moves no predicted probability by more than 2 points.
• Monotonicity. One more month of repayment delay never lowers the predicted risk.
• Absurd inputs. A credit limit of one billion or an age of 200 still yields an output in $[0, 1]$.
• Stability. Five resamples of the training fold give AUCs within 0.01 of each other.
• Slices. No marital-status group falls below an AUC of 0.65.

Each property is both a test in tests/test_robustness.py and a notebook cell that explains it. The tests are frank about their limits: for a client with a one-billion limit, the model outputs a probability of 0.0, which is inside the bounds and still not a sensible answer. In production, the schema contract should reject such a row before the model sees it.

The last cell simulates a drift: every credit limit doubles, and the mean predicted risk falls by 1.4 points. Nobody can say whether that is right, because the model never saw such clients. That is precisely why inputs are monitored after deployment, the subject of the session on drift and conformal prediction.

Back to contents.

Working as a team: branches, pull requests, CI, interface

The main branch is the version that works; every change goes through a short branch, a pull request reviewed by someone else, and a CI that reproduces the numbers.

The cycle the branches guide describes

1. Start from main. Switch to main, pull, and create a short branch named after one task.
2. One commit per idea. Present-tense messages that say what and why, so that git log reads as the project log; never data, models, secrets or notebook outputs in a commit.
3. Pull request. Three lines: what it changes, how to check it, what is left; reviewed by someone other than the author.
4. Merge with a merge commit. Not a squash, so that the history keeps the trace of each branch.
5. Tag the hand-in. The last commit pushed before a deadline counts; a tag such as jalon-1 marks it.

The continuous integration file installs the pinned versions, runs make reproduce and make test, then computes the metrics a second time and compares the two metrics.json. If a single line differs, the check is red and the pull request does not merge. This is exactly the entry criterion of the first milestone: another team clones your repository and gets your numbers.

The interface, app/app.py, is a Streamlit page of about sixty lines: fields for the variables, a button, the probability and the decision. It loads the same Pipeline as the batch and reads the threshold from config.yaml, and it contains no business logic of its own. A computation changed only in the interface would silently diverge from the nightly batch, which does the same thing on a whole CSV file.

Back to contents.

What is left: three milestones, a heavy model, five rules

The repository is a starting point; three milestones complete it, and the final project reuses the same structure on another subject.

The heavy model is deliberately empty: the function in src/credit/model.py lists what to do (gradient boosting in a pipeline with a fixed seed, hyperparameters by cross-validation on the training fold, the same calibration as the baseline, the champion rule written first, a Nadeau and Bengio corrected paired test) and raises an error until it is written. make compare also refuses to run while the decision log lacks the champion rule. Writing a boosting model takes seconds with an assistant; explaining every hyperparameter, and showing the interval that separates the champion from the logistic, is the actual work.

• Notebooks import src/credit and never paste a copy of it.
• All fitting, scaling and encoding use the training fold only, never the full dataset.
• The test fold is read once, by make compare, after the champion rule is written.
• SEX never enters the model; marital status is allowed, with a fairness audit.
• Code may be written with an AI assistant, but each member must be able to explain every line orally.

Four of the five rules have an automatic guardian: the feature function refuses forbidden columns, the Pipeline carries preprocessing, make compare checks the decision log, the tests freeze the split. The fifth is checked in person.

Back to contents.

Key takeaways

1. Version the code, the configuration and the fingerprints; regenerate the data, the folds and the models with one command.
2. A machine learning test checks a property that must always hold, such as a frozen split or a perfect AUC on separable data, not a good score.
3. The decision threshold follows from the error costs and lives in configuration; on these costs it is 600/5,100, about 0.118.
4. A model must beat the rule already in place, in euros and across cost levels, with an interval; here the logistic does so only modestly.
5. Write robustness properties before looking at the numbers, and admit what they cannot settle, such as behaviour under unseen drift.
6. Short branches, reviewed pull requests and a CI that reproduces twice keep the main branch trustworthy.

Glossary

Train / validation / test split

The three disjoint folds: one to fit, one to calibrate and choose the threshold, one read once for the final comparison.

Random seed

The fixed number that makes the split and the training repeatable.

Data leakage

Any use of information from validation or test data, or from the future, while fitting the model.

Base rate

The share of defaults in the data, 22.12% here; the prediction of the naive model.

Brier score

The mean squared gap between predicted probability and outcome; lower is better.

Expected cost

The average cost per file of a decision rule, weighting each error by its price.

Property-based test

A test that checks a property expected to hold whatever the data, rather than a particular score.

Schema contract

The declared columns, types and bounds that every input row must respect.

Continuous integration (CI)

Automatic reproduction and testing of the repository at every push and pull request.

Pull request

A proposed change on a branch, reviewed by someone else before it is merged into main.

Milestone

A dated hand-in, identified by the last commit pushed before the deadline and usually a tag.

Model card

A document stating what a model is for, how it was evaluated, and its limits.

Ablation study

Removing a group of features, pre-registered, to measure what it really contributes.

The other notes of the course

One public summary per deck, in the order of the course.

Previous: Defending a machine learning projectNext: Asset returns: XGBoost and sentiment

1. IntroductionAI winters, jobs and method
2. ToolsKaggle, GitHub, Python: the tools
3. Session 1Project framing and least squares
4. Session 2Maximum likelihood and logistic regression
5. Session 3Gradient descent, ridge and lasso
6. Session 4Decision threshold and reproducible ML
7. HomeworkA defensible baseline model
8. Session 5Kernels, SVM and decision trees
9. Session 6Forests, boosting, honest comparison
10. Session 7SHAP, fairness, PCA and clustering
11. HomeworkA documented champion model
12. Session 8Neural networks, drift, conformal prediction
13. HomeworkA delivered machine learning project
14. Session 9Defending a machine learning project
15. ReferenceCredit default scoring project
16. ReferenceAsset returns: XGBoost and sentiment
17. ReferencePython code for every ML model
18. ReferenceDeep learning essentials
19. ReferenceOpen data ML projects by sector
20. ReferenceData and AI engineer handbook
21. ReferenceData engineering, SRE and cloud
22. ReferencePanorama of machine learning models

All the notes, as a reading list

Attention and transformers, step by step

About these notes

Written by the lecturer from the slides of the deck, as a public summary: the key ideas, the formulas and one worked example. The exercises, their solutions, the tests and the grading stay in the deck, for the students of the course.

Revision: 16 October 2026.