~/teaching · data analytics · session 4, thursday 7 january 2027

Predictive analytics for decision-makers: what models learn, four traps, and the price of each error

Judge a model on cases it has never seen, ask when each feature becomes known, and put a price on every kind of mistake.

Sitraka Forler, Université Catholique de Lille

Revision 16 October 2026 · about 14 minutes

Abstract

This is the fourth session of Data Analytics for Business, a course of the M2 International Management at Université Catholique de Lille (2026-2027). It comes the same day as SQL essentials for managers and climbs one rung higher on the ladder of the first session: from what happened to what will happen. The case is still Olist, a Brazilian marketplace whose order history is openly published.

The audience is future managers, not data scientists, so the first ninety minutes use nothing harder than a ratio. They cover what a model learns, the three families of models and the question each one answers, four traps that make a model look better than it is, and the reading of a confusion matrix as a table of costs. The second half moves to a prepared Google Colab notebook where two real models are compared for one decision, with no code to write, and ends with a one-page memo written in class. The main reference is Provost and Fawcett, Data Science for Business, chapters 5 and 7.

After the session you can tell regression, classification and clustering apart, recognise overfitting, data leakage, a biased sample and a correlation taken for a cause, and turn the errors of a classifier into money to decide whether, and how, to use it.

The slides

Predictive analytics for decision-makers

Session 4, Thursday 7 January 2027. Data Analytics for Business, Université Catholique de Lille, M2 International Management, 2026-2027.

Open the slides (password from your teacher)

The deck itself is encrypted for the students of the course; this page summarises it for everyone.

The course page

Plan of the deck

Opening. Three recall questions from the morning, and a vendor offering a delivery model with a very high accuracy figure: would you buy it?
What a model learns. Past examples, a pattern, an unknown; features and a target; three families of models and three blind spots.
Four traps. Training and test data, overfitting, data leakage, biased samples and correlation taken for causation, each on the case data.
Errors have a price. The confusion matrix, the accuracy trap, precision and recall, a worked cost example and the choice of a cut-off.
Two models, one decision. A prepared Colab notebook: compare two models at the same cut-off and cost table, and find the feature that leaks.
Milestone 3 in class. A one-page memo for a decision owner: what the model predicts, how well, its limits, and the decision.
Wrap-up. Six ideas to keep and the plan for the next day.

What a model learns, and what it cannot know

A model looks at past cases whose outcome is known, finds a pattern, and uses it to guess the outcome of new cases.

There is nothing magical about a predictive model. It is shown many past cases where the answer is known, it finds a regularity linking what was known at the time to that answer, and it applies that regularity to new cases whose answer is still unknown. The deck illustrates the idea with a Galton board: the path of a single ball is unpredictable, the shape of the pile is not.

In a table with one row per order, the columns the model uses to guess are the features, and the column it tries to guess is the target. Two rules follow at once. Every feature must be available at the moment the prediction is made, and the target must be known for the past rows, or there is nothing to learn from. This is also why the row checks of the SQL session matter: a join that duplicated orders makes some of them count twice in training.

Three blind spots

What no column records. A strike, a storm or a seller running out of stock is invisible if nothing in the data captures it.
What has changed. A model assumes the future resembles its past, and the case data stop in 2018: new carriers or payment habits are unknown to it.
Why things happen. A model finds associations that predict; it does not say what causes what.

Back to contents.

Three families of models, chosen by the question

How much, which class, or which groups: the business question fixes the kind of model, not the data available.

**Table 1:** The three families a manager meets most often
Family	Question it answers	Example on the case
Regression	How much?	Days until an order reaches its customer
Classification	Which class?	Late versus on time, bad review versus not
Clustering	Which groups?	Sellers or customers with similar behaviour

Regression predicts a quantity, classification a category. Clustering is different in kind: there is no target, the algorithm groups similar cases, and it is up to the analyst and the manager to decide whether the groups mean anything for the business. Customer segmentation is the classic use.

The same business topic can lead to two families. Asking whether an order will be late is a classification; asking how many days it will take is a regression. The right choice depends on the decision the prediction is meant to support. The deck then has the class vote on a series of short business questions, and asks each group whether its own brief contains a useful prediction. Not every brief needs one: a clean description of the right problem often beats a weak forecast.

Back to contents.

Test data and overfitting

A model must be graded on cases it never saw while learning, because a model can memorise noise and look perfect on its own training data.

The deck compares model evaluation to an exam. If the exam reused the practice questions, a student who learned the answers by heart would score full marks without understanding anything. So part of the data, the holdout or test data, is set aside, never shown to the model while it learns, and used once at the end to grade it. With dated data, a sound habit is to hold out the most recent months, which mimics using the model on the future.

Remark (Principle (holdout)). Score a model on data it has never seen, never on the data it was fitted to.

Every dataset mixes a real pattern with noise, random variation that will not come back. A flexible enough model can bend through every training point; it then looks flawless on the training set and breaks down on new orders, because it has learned the noise as if it were the pattern. This is overfitting.

Provost and Fawcett draw this as a fitting graph: plotted against model complexity, the training error only goes down, while the test error first decreases and then climbs again once noise is being memorised. The model worth keeping sits at the low point of the test curve, not where the training curve looks best. The deck presents the graph as a concept, not as a measurement.

Back to contents.

Data leakage: information from the future

A leak happens when a model relies on something that would not yet be known at the moment the prediction is made.

Suppose the goal is to predict, as soon as an order is approved, whether it will be delivered late. Some columns are available at that moment, others are filled in afterwards. A model that uses the second kind is cheating: it scores brilliantly in testing and fails on the first day of real use.

**Table 2:** Some columns of the case, by when they become known (prediction made at approval)
Column	Known at approval?	Comment
order_estimated_delivery_date	yes	the date promised at checkout
price, freight_value	yes	set when the order is placed
product_category_name	yes	a property of the product
customer_state, seller_state	yes	known from the accounts
order_delivered_customer_date	no	it is the answer itself: late means delivered after the promise
review_score, review_comment_message	no	written by the customer after delivery

The two obvious leaks in the deck sound absurd once said aloud: the delivery date to predict lateness, the review text to predict a bad review. In a table with dozens of columns, they are easy to miss. Subtler leaks hide in fields that are updated after the event; in the case file, the order status records the latest state of each order, not its state at checkout.

Remark (Principle (too good to be true)). When a model performs far better than expected, look for a leak before celebrating, and ask of every feature: when does this become known?

Back to contents.

Biased samples, and patterns that are not causes

A model only knows the cases it was shown, and a pattern that predicts well says nothing about which way the cause runs.

A biased sample means the data describe only part of the world the decision will apply to. In the case, a review score exists only when a customer answered the satisfaction survey, and a comment only when they also wrote something; those who write may be the most pleased or the most irritated. The data cover one country over about two years. And keeping only sellers with many orders, to stabilise averages, studies the sellers who stayed rather than those who left. A brief should say who is in its data and who is not.

The fourth trap is a correlation read as a cause. The deck gives two hypotheses, explicitly presented as things to test rather than findings. If orders paid in instalments turn out to be larger, pushing instalments only pays if paying in instalments raises what people spend; if large baskets lead people to choose instalments, the push will not grow revenue, and a model that predicts order value from instalments works equally well in both stories. If late orders get lower reviews, distance or product category may drive both lateness and dissatisfaction.

Remark (Compare like with like). Before paying for faster deliveries, put late and on-time orders side by side inside one distance band and one product category. A review gap that persists there makes a far stronger case.

The deck closes this part with practice cases, invented for the exercise, on which the class votes for the trap at work. In the memo at the end of the session, these four traps become the limits a group must report.

Back to contents.

Errors have a price: confusion matrix, precision and recall

A classifier makes two kinds of mistakes, false alarms and misses, and they rarely cost the same amount.

Each prediction of a yes-or-no model ends in one of four boxes of a confusion matrix. Two are right: the late order that was flagged (true positive) and the on-time order left alone (true negative). Two are wrong in different ways: a false alarm, which wastes a contact (false positive), and a miss, which lets a problem through (false negative).

The accuracy trap

Accuracy counts the proportion of correct predictions. When the event is rare, it misleads. In the deck's hypothetical example, nine orders in ten arrive on time, so a model that always answers "on time" scores 90% and is useless. Never accept an accuracy figure without asking for the base rate of the event.

\text{accuracy} = \frac{TP + TN}{TP + FP + FN + TN}

(1)

Two other ratios answer the questions a manager actually asks. Precision: when the model raises a flag, how often is it right, in other words how much effort is wasted? Recall: of all the real problems, how many does it catch, in other words how much still slips through?

\text{precision} = \frac{TP}{TP + FP}

(2)

\text{recall} = \frac{TP}{TP + FN}

(3)

Worked example from the deck (the sample exam question of the syllabus)

The situation. The model marks 200 orders as at risk of a negative review; 120 of these actually get one, and 80 further orders with a negative review go unflagged.
Fill the boxes. TP = 120, FP = 200 − 120 = 80, FN = 80. The true negatives are not given, and they are not needed: those orders cost nothing and earn nothing.
Precision. 120 / 200 = 0.60: six flags in ten point at a real problem.
Recall. 120 / (120 + 80) = 0.60: the model catches six problems in ten.
A coincidence. Both ratios equal 60% here only by chance; they answer different questions and usually differ.

The deck then attaches money to the boxes: a contact costs €2 in advance and prevents the negative review, and each negative review avoided is worth €10. Only the flagged orders are contacted, so the net value of using the model is the value of the problems caught minus the cost of all contacts. Students compute it in class to decide whether to use the model, then redo it with only the price of a contact changed, which shows that the decision can flip while the model stays the same. This is the expected-value reasoning of Provost and Fawcett's chapter 7.

\text{net value} = TP \times \text{value per problem avoided} - (TP + FP) \times \text{cost per contact}

(4)

Choosing the cut-off

A classifier seldom answers yes or no: it scores every order, and a person sets the cut-off beyond which the company acts. Lowering it flags more orders, so recall rises while precision usually falls and more contacts must be paid for; raising it does the opposite and misses more problems. Which balance is right is not a technical question. With the deck's prices, contacting an order with risk $p$ is worth it when the expected gain exceeds the cost:

p \times 10 > 2 \iff p > \tfrac{1}{5}

(5)

This rule assumes the score can be read as a probability, which the team that built the model should confirm. Each group then decides, for its own brief, whether a false alarm or a miss is the more expensive mistake, and who pays for each.

Back to contents.

Comparing two models for one decision

Two models are compared on the same test data, at the same cut-off and with the same cost table, and doing nothing is a legitimate answer.

The hands-on part runs in a prepared Google Colab notebook. Students change settings in form fields and rerun the cells; they never touch code. They read the outputs in the order of the session: which features each model uses, the four outcomes on the test data, precision and recall, then the money.

**Table 3:** The comparison grid each group fills in
Line	Model A	Model B
Features used	…	…
Precision on the test data	…	…
Recall on the test data	…	…
Net value at the agreed costs	…	…
Leaking feature?	…	…

Start by putting the two models on one cut-off and one cost table; with different settings you would be comparing two decisions, not two models.
Then move the cut-off and watch the net value, and note the one you would choose and why.
One of the two models relies on a leaking feature: find it by asking, for each feature, when it becomes known.
"Neither" is a valid decision, provided it is argued.

When two groups pick different models from the same numbers, they have usually priced the errors differently, which is the point of the afternoon. The session ends with a one-page memo for a decision owner who has not seen the notebook: what the model predicts, how well (precision, recall and net value, with the model and cut-off named), the limits that owner must know, and the decision with what would change it. A strong memo gives numbers a manager can check, states a limit plainly and proposes an action, such as a test in one region, rather than quoting a single accuracy figure.

Five questions to ask before buying a predictive model

Base rate. What is the base rate of the event, and how would a model that never predicts it score?
Holdout. Was it tested on cases it had never seen?
Leakage. Does it use anything that is only known later?
Sample. Which cases was it trained on, and who is missing?
Costs. What are its precision and recall, and what does each kind of error cost us?

Back to contents.

Key takeaways

The question picks the family of model: how much, which class, or which groups.
Judge a model only on data it has never seen; a perfect fit on the training data is a warning, not a result.
When a model looks too good to be true, check when each feature becomes known before celebrating.
Ask who is missing from the data before generalising what a model has learned.
A pattern that predicts is not proof of a cause; compare like with like before funding a fix.
Accuracy hides rare events: read precision and recall, then price each kind of error before choosing a model or a cut-off.

Glossary

Feature: A column a model uses to make its guess; it must be known when the prediction is made.
Target: The column a model tries to guess, known for past cases only.
Regression: A model that predicts a quantity, such as a number of days.
Classification: A model that predicts a category, such as late or on time.
Clustering: Grouping similar cases without a target; the meaning of the groups is for people to judge.
Holdout data: Data set aside while the model learns and used once to grade it.
Overfitting: Learning the noise of the training data, which looks perfect there and fails on new cases.
Data leakage: Using information that would not yet be available when the prediction is made.
Biased sample: Data that cover only part of the population the decision will apply to.
Confusion matrix: The four-box table of true and false positives and negatives.
Accuracy: Proportion of correct predictions; misleading when the event is rare.
Precision: Share of the flagged cases that are real problems.
Recall: Share of the real problems that the model flags.
Cut-off: The score above which the business acts on a prediction; set by the costs of errors.

The other notes of the course

One public summary per deck, in the order of the course.

Session 1From data to decisions
Session 2Data visualisation and storytelling
Session 3SQL essentials for managers
Session 4Predictive analytics for decision-makers
Session 5Generative AI, GDPR and the AI Act
Session 6Pitching data insights

All the notes, as a reading list

About these notes

Written by the lecturer from the slides of the deck, as a public summary: the key ideas, the formulas and one worked example. The exercises, their solutions, the tests and the grading stay in the deck, for the students of the course.

Revision: 16 October 2026.