Open data and machine learning projects by sector: finance, insurance, logistics, tech, sport

Where to find open data, family by family

Open data comes in five families, from cleaned tables with a ready target to raw public series that you must clean and join yourself.

The deck sorts its sources from the most ready to use to the rawest. Machine learning platforms give cleaned tables with a target: convenient, but thousands of people have already worked on them. Public statistics give clean official series, rarely with a target. Macro and financial APIs give rates, indices and prices. Real-world sources (news, weather, maps, encyclopaedic facts) seldom make a project on their own but enrich any of them. Sport gives very detailed match events.

The trade-off is between comfort and originality. A Kaggle table loads in one line, and your project will look like a thousand others; joining two public sources costs a week of cleaning and gives something nobody has done. The deck recommends the middle path that produced the best projects so far: a clean starting dataset, enriched with one outside source such as the weather of the day added to store sales.

Back to contents.

Reading a data licence, and what GDPR still forbids

Two questions settle almost every licence: may I publish something derived from the data, and may I use it commercially?

A derivative is anything built from the data: a cleaned file, a chart, a trained model. The deck reduces the licences met in the catalogue to those two questions, which is enough for coursework. A third one matters for production and remains legally open for non-commercial licences and for corpora collected on the web: does the licence also cover the model trained on the data? When in doubt, the honest move is to ask the producer.

A permissive licence never lifts the General Data Protection Regulation (Regulation (EU) 2016/679). Data that can be linked to a person, directly or by cross-referencing, is personal; replacing names with codes (pseudonymisation) does not change that; health, opinions or origin are special categories under Article 9. The French DVF file of property sales is open, yet its terms forbid re-identifying sellers or letting search engines index the sales.

• Work on aggregated data, data anonymised by its producer, or synthetic data (Synthea for patients, PaySim for mobile payments, IBM's simulated laundering transactions).
• Do not scrape profiles from social sites or review platforms: a school project has neither the legal basis nor the means to inform the people concerned.
• Push no person-level data to a public repository, even pseudonymised, and keep API keys in a .env file that Git ignores.

Back to contents.

The machine learning project brief in six lines

Before any code, a team answers six questions in one sentence each; a line it cannot fill this week means the topic must change, not the model.

The template

1. Decision. Who acts on the prediction, and what each kind of error costs.
2. Target. What is predicted, at which horizon, and on which date it becomes known.
3. Metric. The one that measures the decision, with the level at which the model starts to be useful.
4. Safeguards. Forbidden variables, protected groups, possible leaks from the future.
5. Data. Address, licence, period covered, and what the data does not contain.
6. Plan. Three models (naive, best reasonable, heavy), the milestones, and who does what.

The decision comes first because it fixes the rest: the useful target, the relevant metric, the errors that hurt. A project with no identifiable decision-maker ends as a demo with nothing at stake. The brief is a file in the repository, approved and amended through pull requests, as set up in the tools note; the first homework builds the baseline model on it.

A filled-in example: late deliveries on Olist

The target line does the most work: it states when the outcome is known and when the prediction is made, so everything that happens in between is banned as a feature. The licence line matters too: non-commercial and share-alike suits coursework, not resale.

Three mistakes seen every year

• A vague target. "Satisfaction" is not a column. A target is a rule you can apply to every row, with a date, for instance a low rating given within a month of delivery.
• The wrong metric. On the ULB card fraud data, 492 frauds among 284,807 transactions ($492/284\,807 \approx 0.173\%$, about one in 579), a model that always says "no fraud" reaches 99.83% accuracy and stops nothing. Measure what the analysts can catch with the alerts they can review.
• Data from the future. The number of payment reminders predicts default very well, because reminders follow the default. This is the most common form of data leakage; the cure is to ask, for each variable, on which date it is known.

Back to contents.

Machine learning project ideas by sector

Each sector has a typical decision, an open dataset to start from and a trap of its own; the deck writes one complete project sheet per main sector.

A project sheet uses the brief's grid plus three boxes: what the team delivers, with which packages, and why anyone would pay for it. The table below gathers the ten ideas the deck judges feasible in three months; together they cover the problem types of the course (classification, count and amount regression, time series, ranking).

Finance: credit, fraud, market risk

A credit project follows a loop: features known on the application day, a twelve-month probability of default, two cost-based thresholds that split accept, review and reject, then retraining once outcomes are known. Its built-in flaw is selection bias: default is only observed for accepted applicants, and every public dataset contains granted loans only. Age and sex are left out of the features but kept aside to measure fairness, and credit scoring of individuals is a high-risk system under Annex III of the EU AI Act (Regulation (EU) 2024/1689). For card fraud, the ULB data cover two days: train on the first, test on the second, judge with the precision-recall curve and with recall at a fixed number of alerts, reweight classes before trying resampling, and recalibrate afterwards. In market finance, returns are hard to predict (the asset price note shows why), volatility much less so because it clusters, and a 99% value at risk is checked by counting the days it is exceeded: about 2.5 are expected in 250 days, and the Kupiec test (1995) says whether the gap is chance.

Insurance: frequency times severity

Non-life pricing multiplies two models. Frequency, the number of claims per year, is a Poisson GLM with a log link in which the exposure $e_i$ (the insured period, in years) enters as an offset, so that six months count as half a year; severity, the cost of a claim, is a Gamma GLM fitted on policies with claims. The reference dataset, freMTPL2, is a French motor portfolio with 678,013 policies and 26,639 claim amounts (sizes read on OpenML). Sex may not change a premium in the EU since the Test-Achats ruling (CJEU, 2011). The study of the German actuarial association (DAV, 2024) and Henckaerts et al. (2021) find boosting ahead of the GLM; a common practice is to let boosting reveal interactions, then add them to a GLM that a pricing committee can defend.

Logistics: the model predicts, the solver decides

Machine learning forecasts quantities and times; operations research turns them into orders and routes. In the M5 competition (Walmart sales, 42,840 hierarchical series), learning methods led by LightGBM dominated for the first time, the winner beating the best statistical benchmark by 22.4% according to Makridakis, Spiliotis and Assimakopoulos (International Journal of Forecasting, 2022). The scaled error RMSSE reads simply: below 1, you beat the naive forecast. A 90% quantile forecast sets a safety stock directly. Vehicle routing is an optimisation problem solved with OR-Tools, not a learning one, and saying so shows you understand the trade. On the NASA C-MAPSS engines, predict the remaining useful life and split by engine, never by row.

Big tech: recommendation, ranking, experiments

Recommendation fills a nearly empty users by items matrix by giving each user and each item a short vector of learned tastes; real systems see clicks rather than ratings, hence alternating least squares on implicit feedback (the implicit library). On MovieLens, split by time, and expect "most popular" to be hard to beat on precision at ten; report catalogue coverage too. Search ranking is scored with NDCG@10 on Amazon's ESCI queries; click prediction for advertising with log-loss, because the bid uses the probability itself. Moderation models on the Jigsaw comments over-score identity words, so error is measured per group. In every case the offline metric is a promise that only an A/B test can keep.

Football: expected goals and results

An expected goals model gives each shot its probability of scoring from distance, angle and context; summed, the probabilities give a team's or a player's xG. Because they are summed, calibration matters more than ranking, so use log-loss, Brier score and a calibration curve, split by match, and treat penalties apart. To predict results, the attack and defence Poisson model of Maher (1982) and Dixon and Coles (1997) is the classic; the real benchmark is the bookmaker's odds, turned into probabilities $1/c$ and rescaled to remove the margin. The project measures probabilities and recommends no bet; Transfermarkt's terms forbid model training and football-data.co.uk reserves its files for private use.

Seven other sectors in one line each

• Energy. Next-day national consumption from RTE's éCO2mix (Licence Ouverte) and Open-Meteo temperatures; use the weather forecast, not the weather observed afterwards.
• Real estate. Price per square metre from DVF; train on 2021 to 2024, test on 2025, publish aggregates only.
• Health. Synthetic Synthea records, since real data such as MIMIC-IV is for credentialed researchers; a risk score does not say which treatment works.
• Human resources. Attrition on IBM's fictional HR dataset; recruitment is high risk under Annex III, and Reuters (2018) reported that Amazon dropped a CV screener that penalised the word "women's".
• Public sector. Severity of road accidents from the French injury accident files; audit targeting systems have been overturned in court, as SyRI was in the Netherlands in 2020.
• Marketing. Churn on IBM's Telco data, then uplift: whom an offer actually keeps, which needs randomised data such as Hillstrom's 2008 e-mail test and is judged with a Qini curve.
• Industry. MVTec AD images and SECOM sensors; learn what a normal part looks like and flag deviations, at a fixed false reject rate.

Back to contents.

Repository structure and Python packages for an ML project

Every project uses the same repository layout; only the data README and the brief change from one sector to another.

• README.md at the top: what the project does, for whom, and how to run it.
• docs/ holds the six-line brief and a journal of dated decisions.
• data/README.md records the producer, address, licence, download date, size, and what the data lacks. The data itself is fetched again by a preparation script, which is mandatory for competition and personal-use data and prudent everywhere.
• notebooks/ for exploration, src/ for data, features and models, batch/ for the prepare, train and evaluate scripts.
• app/ for a Streamlit interface, tests/ for leakage, target and split checks, a config file for seeds, thresholds and dates, and pinned versions in requirements.txt.

The naive model is not a formality: it is the reference that gives every other number its meaning, and it should look like what the business does today. The models overview details each family and its pitfalls.

Back to contents.

What a machine learning project pays, and compared with what

A gain is counted against what the company already does, never against having no model at all, and every assumption is either sourced or labelled.

The deck gives two ways to put a figure on a gain. The first counts errors avoided: $V$ decisions, the change $\Delta$ in false negatives and false positives, and the unit cost $c$ of each. In credit a false negative is an accepted default, a false positive a good customer turned away; in fraud a false positive is an analyst's wasted time or a card blocked by mistake. The second counts time: $N$ cases a year, $\Delta t$ hours saved per case, $w$ the fully loaded hourly cost. Costs are then subtracted.

A worked example on a fictitious bank (every assumption invented by the deck)

1. Frauds. One million card transactions a month at a 0.2% fraud rate give 2,000 frauds a month.
2. Extra frauds stopped. Rules stop 40% of them, the model 60% with the same number of alerts: 2,000 × 0.2 = 400 more a month.
3. Gross gain. At an average loss of €80 per fraud, 400 × 80 = €32,000 a month, or €384,000 a year.
4. Net gain. Maintenance at €3,000 a month costs €36,000 a year, which leaves €348,000 net.

In a real project the fraud and detection rates come from the test set with a confidence interval, the average loss from a public source or from the data (the ULB file has every amount), and the maintenance cost from an estimate you explain. An interval on the gain is better than one number.

Back to contents.

How to choose a machine learning project

A good topic earns four yeses: the data downloads this week, the decision is clear, three models can be compared, and a simple page can show one case.

The selection grid

1. Data this week. An address, a licence and a download that works now. This one is eliminatory: a download that fails today will not work better in two months.
2. A clear decision. Someone could do something different because of the prediction.
3. Three comparable models. A naive one, a best reasonable one and a heavy one, on the same split and the same metric.
4. A feasible interface. A page, for instance in Streamlit or Gradio, where one enters a case and reads the answer, with no server to run.

The recipe behind the strongest projects of past editions is short: a clean dataset from the catalogue, plus one outside source that adds an idea (the weather when forecasting demand, the context of each shot for expected goals, road distances for late parcels), plus a decision you can explain to a non-specialist.

What kills a project, model aside

• Missing data: promised for next week, never delivered, or delivered without the target. The course cites Gartner (February 2025) predicting that 60% of AI projects lacking AI-ready data will be abandoned by 2026.
• A fuzzy target: no one is able to state which rows are successes.
• No owner: the prediction has no business team willing to act on it.
• Procurement: security reviews, contracts and purchasing that take longer than the project itself; working on open data is how the course sidesteps it, and the course introduction returns to it.

The team project is free in its topic but follows the method of the common thread: a brief, a reproducible repository, three models compared honestly, a model card, slides and an interface. The homework notes describe the steps, from the baseline to the champion model and the final delivery.

Back to contents.

Key takeaways

1. Choose the data before the model: if you cannot write down an address and a licence this week, change the topic.
2. Two questions read most licences, publishing a derivative and commercial use, and no licence exempts you from GDPR.
3. The six-line brief starts with the decision; the target line must say when the outcome is known and when the prediction is made.
4. Each sector has its trap: selection bias in credit, a rare class in fraud, exposure in insurance, the solver in logistics, the A/B test in tech, tiny samples in football.
5. Three models every time, and a naive one that resembles current practice; the heavy model stays only if it changes a decision.
6. A gain is measured at the margin over what already exists, with sourced or labelled assumptions, and net of data, compute, people and maintenance.

Glossary

Open data

Data published so that anyone may reuse it under a stated licence, such as CC BY 4.0, CC0, the French Licence Ouverte or ODbL.

Scope document (project brief)

The six lines a team writes before coding: decision, target, metric, safeguards, data and plan.

Data leakage

Information in a feature that would not be available at the moment of the decision; it flatters the test score and fails in production.

Baseline

The naive model a project must beat, ideally close to what the business does today.

Share-alike

A licence clause (CC BY-SA, ODbL) requiring derived databases to be published under the same licence.

Pure premium

Expected claim frequency times expected claim cost for a profile, before expenses and margin.

Exposure

The time a policy was insured, in years; it enters a frequency model as an offset.

Precision-recall curve

Precision against recall across thresholds; more telling than the ROC curve when positives are rare.

Value at risk (VaR)

The loss exceeded with a given small probability over a horizon, for example one day in a hundred at 99%.

Vehicle routing problem (VRP)

Assigning stops to vehicles and ordering them under capacities and time windows; an optimisation problem solved by heuristics.

Matrix factorisation

Approximating a sparse users by items matrix by the product of two small matrices of latent tastes.

Adverse selection

Good risks leave for a better-priced competitor and bad risks stay, the cost of an inaccurate tariff.

Anomaly detection

Learning what normal data looks like and flagging deviations, used when defects are too rare to learn directly.

The other notes of the course

One public summary per deck, in the order of the course.

Previous: Deep learning essentialsNext: Data and AI engineer handbook

1. IntroductionAI winters, jobs and method
2. ToolsKaggle, GitHub, Python: the tools
3. Session 1Project framing and least squares
4. Session 2Maximum likelihood and logistic regression
5. Session 3Gradient descent, ridge and lasso
6. Session 4Decision threshold and reproducible ML
7. HomeworkA defensible baseline model
8. Session 5Kernels, SVM and decision trees
9. Session 6Forests, boosting, honest comparison
10. Session 7SHAP, fairness, PCA and clustering
11. HomeworkA documented champion model
12. Session 8Neural networks, drift, conformal prediction
13. HomeworkA delivered machine learning project
14. Session 9Defending a machine learning project
15. ReferenceCredit default scoring project
16. ReferenceAsset returns: XGBoost and sentiment
17. ReferencePython code for every ML model
18. ReferenceDeep learning essentials
19. ReferenceOpen data ML projects by sector
20. ReferenceData and AI engineer handbook
21. ReferenceData engineering, SRE and cloud
22. ReferencePanorama of machine learning models

All the notes, as a reading list

Attention and transformers, step by step

About these notes

Written by the lecturer from the slides of the deck, as a public summary: the key ideas, the formulas and one worked example. The exercises, their solutions, the tests and the grading stay in the deck, for the students of the course.

Revision: 16 October 2026.