~/teaching · machine learning · homework 3

Delivering a machine learning project: repository, tested service, traceable report and the documents a committee reads

The last step of a project: the same method as the baseline and the champion, applied to your own topic and shipped

Sitraka Forler, Centrale Méditerranée

Revision 16 October 2026 · about 10 minutes

Abstract

A delivered project is more than a good model. It is a repository that another team can run without calling you, a service that answers a request, a report in which every number comes out of the code, and a set of short documents that a decision-maker and a validation team can read. This note summarises the final homework brief of the Machine Learning course at Centrale Méditerranée (option DDEFI, 2026-2027), where each team applies the method of the course to a topic of its own choice.

The method was learnt on a common thread, a credit default score, in two earlier steps: a defensible baseline and a documented champion model. The final delivery reuses the same repository template and adds what the last sessions taught: a neural network, drift monitoring, conformal prediction, a profit curve, an EU AI Act gap analysis and a prediction service.

What follows is the method, not the class arrangements. It reads as a checklist for anyone handing over a model to people who will run it, sign for it and watch it in production.

The slides

Homework after block 3: the project delivered

Homework 3. Machine Learning, Centrale Méditerranée, option DDEFI, 2026-2027.

Open the slides (password from your teacher)

The deck itself is encrypted for the students of the course; this page summarises it for everyone.

The course page · Attention and transformers, step by step

Plan of the deck

Where the project stands. Two projects, one method: the common thread taught it, your topic applies it.
What a delivery contains. Seven lines: scope, repository, three models, documents, report, interface, individual accounts.
The final repository. The template of the common thread, with a service, an interface and integration tests.
The report. Eight pages at most, ten sections in a fixed order, every number traceable.
The documents. Model card, datasheet, monitoring plan, AI Act gap, client memo, decision log, AI appendix.
The client memo and the API contract. One page in euros for a decision-maker, and a typed contract at the door of the model.
Organisation. Three roles and work signed by name.
Pitfalls and production. Four recurring mistakes, and the circuit a model goes through in a bank.

What "delivered" means

A project is delivered when people who did not build it can rerun it, query it, read why it was built that way and decide whether to use it.

The delivery starts from a scope document agreed early, through a pull request, and kept up to date the same way: the decision the model informs, the target, the metric, the guard-rails, the data and the plan. Three models are then compared under the rules of the champion step, a naive one, a better one and a heavy one, with confidence intervals and a champion rule filed as an issue before the results.

A public repository where make reproduce, make compare and the tests run, with green continuous integration and a README another team has followed alone.
A service and a minimal web page where one enters a case and the model answers.
A report of eight pages at most, and a short slide deck.
The documents of the delivery, complete rather than drafted.
For each member, a one-page account of what they wrote and what they checked themselves, and how.

Remark (Same template, new data). Nothing here is new in kind. Every deliverable was first written on the common thread; the delivery rewrites it for a different problem, which is exactly the skill an employer is hiring for.

Back to contents.

A short report where every number has a source

The report follows a fixed outline so that nothing awkward gets left out, and each number in it can be found in a file produced by the code.

The outline is imposed and identical for all teams, which makes reports comparable and forces the sections people tend to skip. A section can be a single paragraph when there is little to say, but none may be missing.

**Table 1:** Table 1. The ten sections of the report, in order, and what each one answers.
§	Section	Question answered
1	Framing	Which decision, for whom, judged by which metric
2	Data and leakage	Where the data come from and what is known when
3	Methods	Which models were tried, from simple to heavy
4	Comparison	Which model wins, with intervals and a corrected test
5	Threshold and cost	Where to cut, given the cost of each error
6	Conformal prediction	How sure the model is, case by case
7	Fairness	Whether groups are treated differently
8	Monitoring	What is watched in production, and who acts
9	Value	What the model is worth in euros
10	Limits	Where it should not be trusted

The rule that holds it together: every number in the report cites the file and key it comes from, for instance an entry of compare.json. A number typed by hand cannot be checked, so a reviewer will treat it as wrong until shown otherwise.

Back to contents.

The documents a committee reads

Short documents, each with one job, let a reader who will never open the code decide, validate and monitor the model.

Model card: metrics with intervals, threshold, fairness results, and latencies measured on a named machine.
Datasheet: origin, collection, licence, and above all what the data do not cover.
Monitoring plan: drift indicators such as PSI and the Kolmogorov-Smirnov statistic, a justified alert threshold for each, the action it triggers, the person who acts, and a retraining policy.
AI Act gap analysis: two pages comparing the repository with the technical documentation required by Annex IV of the EU AI Act, when the topic falls under it, or a paragraph explaining why it does not.
Client memo: one page in euros for a decision-maker (next section).
Decision log and AI appendix: dated entries up to the last commit, and at least two AI proposals proven wrong.

\mathrm{PSI} = \sum_{b} \big(a_b - e_b\big)\,\ln\frac{a_b}{e_b}

(1)

Here $e_b$ and $a_b$ are the shares of cases in bin $b$ at training time and in production. The formula is the easy part. A monitoring plan earns its keep when, for each indicator, it says from which value something happens and who decides; a list of indicators with no threshold and no owner is the third pitfall below. For the AI Act, read the consolidated text of the regulation rather than slides, since its application dates have already moved once.

Back to contents.

The client memo: one page, in euros

The memo is for someone who will not read the report and must answer yes or no, so it gives the decision, the money, the assumptions and the limits on one page.

The value of a threshold $t$ adds up what the decision earns on true positives and subtracts the cost of each kind of error. On the common thread it is a binary decision; each team adapts it to its own topic. Read at the chosen threshold on the validation set and scaled to a yearly volume, it gives the number a decision-maker asks for first.

V(t) = \sum_i \Big[ g_i\,\mathbf{1}\{s_i \ge t,\, y_i = 1\} - c_{FP}\,\mathbf{1}\{s_i \ge t,\, y_i = 0\} - c_{FN}\,\mathbf{1}\{s_i < t,\, y_i = 1\} \Big]

(2)

Five zones, in this order

The decision requested. One sentence: deploy this score, at this threshold, on this scope.
The gain. What the model earns per year compared with having no model, read off the profit curve.
The assumptions. Cost of each error, threshold, volume and base rate, visible so the reader can see what the number depends on.
The verification. The A/B test that will check the promise: size per arm, duration and stopping rule.
The limits. Fairness, drift, how often the model abstains, and what the data do not cover.

Before handing it in, try it on a reader from outside the field. A memo passes when that reader can state the request and the expected gain in a single sentence; otherwise it needs another draft.

Back to contents.

A prediction service is a contract, not a notebook behind a route

A service promises what it accepts, what it returns and how fast, and integration tests check those promises on every push.

The template exposes three routes with FastAPI: /predict, /health and /model-info, and a Streamlit or Gradio page sits in front for humans. The input contract is a typed pydantic schema in which every variable has a type and a domain, so an age of 250 is rejected with a clear message before it reaches the model. Deployment and observability in general are covered in data engineering, SRE and cloud.

Four integration tests worth writing

A business property. On the credit example, the default score must not fall when the payment delay rises, all else equal; a violation is either a bug or a finding to log.
Load equals train. The model loaded by the service predicts exactly as the trained model does.
Abstention. When the conformal prediction set is too wide, the service answers that it does not know instead of guessing.
Identity. The health and model-info routes return the model version and the hash of the training data.

Latency is measured, not claimed: median and 95th percentile over a thousand calls in the development container, written in the model card with the machine that produced them. It is the first question the team hosting the model will ask.

Back to contents.

Four recurring pitfalls, and what production asks

Most failed deliveries fail on traceability and portability rather than on modelling, and each pitfall has a cheap check that catches it.

Pitfall 1 in numbers: a report that does not describe the repository

The report says 0.781. The AUC was typed by hand from an earlier run of the model.
The repository says 0.774. Rerunning the comparison gives this value in the results file.
The gap is 0.007. It exceeds the tolerance of 0.005 that the continuous integration uses to check the stability of results.
The fix. Regenerate the tables from the results file, cite file and key, and diff the two before handing in.

A service that only runs on the author's laptop, with an absolute path or a package outside the lockfile. Another team runs the serve target and the service tests in the development container.
A monitoring plan with no threshold and no owner. For every indicator, a value, an action, a person.
A memo that hides the limits. Read it side by side with the limits section of the report.

In a bank, a scoring model never goes straight from a notebook to a decision. Development produces the code, tests and log. Independent validation, a team that did not build the model, rereads the data, the leakage and the comparison, and must reproduce the numbers. A committee accepts the model with a threshold, a scope and limits. Deployment brings an input contract, a latency and a traceable version, and monitoring lasts for years. The delivery is a reduced version of that circuit, and the oral defence that follows is described in the project defence. In an interview, few people will ask for your AUC; they will ask how it was validated and what happens when drift arrives.

Back to contents.

Key takeaways

A delivery is judged by what other people can do with it: rerun it, query it, read it and decide.
Keep the repository template of the earlier milestones; one command per job and green continuous integration are the entry ticket.
Every number in the report cites the file and key it comes from; a typed number is a wrong number until proven otherwise.
Each document has one reader and one job: model card, datasheet, monitoring plan, AI Act gap, client memo, decision log.
A service is a contract: typed inputs with domains, a business property under test, abstention when unsure, measured latency.
A monitoring plan without thresholds and owners, or a memo without limits, is not finished.

Glossary

Scope document: The agreed statement of decision, target, metric, guard-rails, data and plan.
Model card: A short document stating a model's uses, metrics, limits, fairness checks and latency.
Datasheet: A document on a dataset's origin, collection, licence and gaps.
Drift: A change over time in the inputs or in their link with the target, eroding performance.
Population stability index (PSI): A measure of how far a variable's binned distribution has moved since training.
Conformal prediction: A method that returns prediction sets with a guaranteed coverage rate.
Abstention: The model declining to answer when its uncertainty is too high.
Property-based test: A test that checks a rule holds across many generated inputs.
Profit curve: The value of the decision in money as a function of the threshold.
Continuous integration (CI): Automatic tests and checks run on every push.
Gap analysis: A comparison between what a regulation requires and what the project provides.

The other notes of the course

One public summary per deck, in the order of the course.

IntroductionAI winters, jobs and method
ToolsKaggle, GitHub, Python: the tools
Session 1Project framing and least squares
Session 2Maximum likelihood and logistic regression
Session 3Gradient descent, ridge and lasso
Session 4Decision threshold and reproducible ML
HomeworkA defensible baseline model
Session 5Kernels, SVM and decision trees
Session 6Forests, boosting, honest comparison
Session 7SHAP, fairness, PCA and clustering
HomeworkA documented champion model
Session 8Neural networks, drift, conformal prediction
HomeworkA delivered machine learning project
Session 9Defending a machine learning project
ReferenceCredit default scoring project
ReferenceAsset returns: XGBoost and sentiment
ReferencePython code for every ML model
ReferenceDeep learning essentials
ReferenceOpen data ML projects by sector
ReferenceData and AI engineer handbook
ReferenceData engineering, SRE and cloud
ReferencePanorama of machine learning models

All the notes, as a reading list

Attention and transformers, step by step

About these notes

Written by the lecturer from the slides of the deck, as a public summary: the key ideas, the formulas and one worked example. The exercises, their solutions, the tests and the grading stay in the deck, for the students of the course.

Revision: 16 October 2026.