The data and AI expert handbook: scope, tool up, model, deploy, observe

The scope document: six lines before any code

Before writing code, write six sentences: the decision the model serves, what it predicts, how success is measured, what is forbidden, which data, and the plan.

The handbook opens with scoping because a model that serves no decision is useless, however accurate. The template has six lines, one sentence each: the decision (who acts on the prediction and what each kind of error costs), the target (what is predicted, at which horizon, and when the truth becomes known), the metric (the one that reflects the decision, with a usefulness threshold), the guardrails (forbidden variables, protected people, leakage), the data (source, licence, period, gaps) and the plan (three models, milestones, who does what). A line that will not fit in one sentence means the problem is not yet understood.

A practical filter follows from it: if a team cannot fill the data line this week, with an address and a licence, it should change the subject rather than the model. The document lives in the repository as Markdown and is only changed through a pull request, so its history shows how the team's understanding matured. A dated decision log sits next to it.

The filled example: default of French SMEs

1. Decision. An account manager reviews the riskiest 5% of small firms first; a false alarm costs one review, a missed default costs a loss.
2. Target. An insolvency notice published in the BODACC within twelve months of the scoring date, which acts as the pivot: every feature must be known on that day.
3. Metric. PR-AUC, recall inside the top 5% and expected cost per file, because the class is rare and the review budget is fixed.
4. Guardrails and data. No data on directors, who are natural persons; BODACC and the Sirene register from data.gouv.fr under Licence Ouverte 2.0; a split by date.
5. Plan. Default rate by sector as the naive model, then logistic regression, then LightGBM.

Six questions put the same template in the sponsor's language: what will you do differently with the prediction, what does each error cost, when will you know the truth, what do you do today without a model, who may see the data, and what would count as failure. The fourth one hands you the naive benchmark for free; the sixth sets the usefulness threshold before any result exists, so it cannot drift towards whatever score comes out.

Back to contents.

The toolbox: eight families, one tool per layer

A data and AI expert works with a stack of tools, from the laptop to the cloud, and a student project needs only one simple tool per layer.

The toolbox is drawn as layers that rest on each other, read from the bottom: the workstation (Python, VS Code, Jupyter, Git, GitHub, Linux, Docker), data (PostgreSQL, DuckDB, SQLite, pandas, Polars, Spark, Kafka, the Parquet format), orchestration (Airflow, dbt, Prefect, LangGraph, MCP), models (NumPy, scikit-learn, XGBoost, LightGBM, PyTorch, TensorFlow, Hugging Face, MLflow), serving (FastAPI, uvicorn, Streamlit, Gradio, Nginx), infrastructure (Kubernetes, Terraform, the big clouds, two French providers, GitHub Actions) and observability. Generative AI is drawn as a column across every layer: it helps write code, extract fields from text, summarise an incident or drive an agent.

Every sheet separates software licence from service price. In machine learning the libraries are free; the cost is in machines and data. A few licences deserve attention: Terraform moved in 2023 to the Business Source License, which is not free software, and the community fork OpenTofu stayed under MPL 2.0; Elasticsearch offers a choice between AGPL v3, SSPL and the Elastic License, while OpenSearch remains under Apache 2.0; Docker Desktop has its own commercial terms, distinct from the free Docker engine. Prices and free-tier quotas are never copied, because they change; the deck says to read them on the official page the day you sign up, and to set a budget alert before starting any cloud machine.

Kubernetes and Spark are presented as what they are: tools for heavy traffic or data beyond one machine, worth knowing for an interview, not worth installing for a three-month project. Built twice, the same project costs either setup time (free software on a laptop) or a bill that grows with usage (hosted services). Clusters and site reliability are covered in the data engineering and cloud note.

Back to contents.

Airflow, and orchestrating language-model agents

An orchestrator runs tasks in the right order at the right time, retries those that fail, and remembers every run, whether the tasks move data or call a language model.

Airflow describes a pipeline as a DAG (directed acyclic graph): tasks and the dependencies between them. It launches the graph on a schedule, records the state and logs of each task for each run, retries a failed task according to a rule, then alerts the team. Scheduling alone is something cron can do; what makes Airflow valuable is that memory, which lets you see which task failed on which day and replay a past day (a backfill) without rerunning everything.

The same machinery runs agents. An agent in a company is usually a chain of language-model calls with precise instructions, placed between ordinary steps. The deck's example is a press watch for a bank: collect articles, summarise them, classify each as a risk or not, check the output with rules and a judge model, wait for an analyst's approval, publish in the morning. To Airflow each call is just a Python function, and the orchestrator supplies what hand-written agent scripts lack: retries on provider errors, a log of every call for audit, cost per task, and an explicit human step before anything irreversible.

Back to contents.

Seventeen model sheets, from simple to deep

Every model gets the same one-page sheet: the idea in a sentence, the formula, a sketch, when to use it, the settings that matter and the function to import.

The sheets come in three tiers. Simple: the mean or last value (persistence), ordinary least squares, logistic regression, k-nearest neighbours, naive Bayes. Intermediate: the decision tree, the random forest, gradient boosting and XGBoost, SVMs with kernels, PCA, k-means. Deep: the multilayer perceptron, convolutional and recurrent networks, the transformer, the large language model, then fine-tuning and RAG. The left half of a sheet is what you must understand, the right half what you must be able to do. A few formulas carry most of the course:

Least squares projects $y$ onto the span of the columns of $X$ (see the least squares note); logistic regression passes a linear score through an S-curve and is the model behind bank scorecards, with a threshold set from error costs rather than left at 0.5. Boosting adds small trees in series, each fitted to what the previous ones got wrong, scaled by a learning rate $\eta$, while early stopping on a validation set picks how many trees to keep. Attention lets each word build its representation from all the others, which the attention article derives step by step.

• On medium-sized tables, boosted trees still usually beat neural networks; an MLP enters the comparison as one more candidate under the same rule.
• A language model produces plausible text, not true text: use it where the output can be checked, such as extracting an amount printed in the document.
• RAG retrieves the closest passages by cosine similarity of embeddings and leaves the weights unchanged; fine-tuning (or LoRA, a low-rank correction $W' = W + BA$) changes the model. RAG for knowledge that changes, fine-tuning for behaviour.

Back to contents.

Which model to choose: naive, best, heavy

Every project compares three models of growing complexity, and climbs to the next one only if the gain is larger than the noise.

The starting point depends on the data and the task, not on fashion. For each kind of problem the handbook proposes one candidate per step: a naive model that gives the scale, a best model that is simple and defensible before a committee, and a heavy model that tests whether complexity pays. The point is a fair comparison, not a victory for the heavy model: one split, one metric, one test period for all three. The models overview gives the long version of this table.

Generative AI projects keep the three steps provided they share a measure: a reference set of questions or documents labelled by the team and frozen before the first attempt, a rubric written in advance, and a cost in euros and seconds per request, because a model two points better and ten times more expensive is not automatically the right one. The deck's own calculation shows why the size of the reference set matters.

Worked example: how precise is a rate measured on 50 questions

1. Observed rate. A model answers 40 of 50 reference questions correctly, so $\hat p = 0.80$.
2. Standard error. $\sqrt{0.80 \times 0.20 / 50} = \sqrt{0.0032} \approx 0.0566$.
3. 95% half-width. With the normal approximation, $1.96 \times 0.0566 \approx 0.111$, about 11 points either side.
4. Reading. The interval runs from roughly 69% to 91%, so a rival scoring 84% on the same set cannot be told apart: say so, or enlarge the set.

Back to contents.

Deploying a model and an application

Deployment turns a notebook into a small service that anyone can call: a script that trains, an API that predicts, a page for humans, and a robot that checks everything at each change.

The architecture is the same for every team; only the model and the interface change. On the developer side you explore in a notebook, move the code that works into src/ with its tests, open a pull request, and GitHub Actions checks style, runs the tests and builds the image. On the host side, an API serves the model and an interface calls it. Secrets stay outside the repository; observability watches the running API.

Step by step

1. Notebook to script. Each step becomes a function with arguments and one command retrains, writing the model file and a metrics file; nobody copies a figure by hand, and the same seed gives the same result.
2. Script to API. FastAPI loads the saved model once at start-up and never retrains; a pydantic class is the input contract, so an out-of-domain request is refused with a 422 error before reaching the model; /docs gives a test page for free.
3. API to interface. A Streamlit page calls the API instead of loading the model, so people and machines see identical predictions; the API address comes from an environment variable, and a visible disclaimer says what the score is not.
4. Tests. Three tests beat zero: the service answers, a probability stays between 0 and 1, an absurd input is rejected. The most valuable one checks that the served model predicts exactly like the evaluated one on a few fixed rows.
5. Continuous integration. A short YAML workflow reruns install, lint, tests and the Docker build on a clean machine at every push and pull request.
6. Container and versions. A Dockerfile installs pinned requirements before copying the code, so the cache survives code changes; exact versions mean the computation in March matches the one in November.
7. Secrets and hosting. API keys live in a local .env ignored by Git and in the host's variables; a key pushed even briefly is revoked at the provider, since deleting it in a later commit leaves it in history. Free hosts suit a demo, but a sleeping instance must be woken before presenting.

The deck insists that an application is software, so five minimum practices apply: lint with ruff, tests in CI, no merge without review, a git tag per milestone with pinned requirements, and a dated decision log. Generative AI speeds up writing an API skeleton, a test or a Dockerfile, but it makes confident mistakes (a package that does not exist, leakage hidden in a preparation step, a key in plain text); the time saved only counts if someone who understands the code reviews it, and each team declares its use. The model serving and drift material is in the neural networks and drift note.

Back to contents.

Observability: logs, metrics, traces, and what to watch on a model

Observing an application means being able to tell that it is slow, failing or drifting before a user complains, from three signals: logs, metrics and traces.

Logs say what happened, event by event, and are useful only when structured: one JSON line per event with named fields (route, status, duration, model version, output), never personal data in clear. Metrics count and measure over time and drive alerts; a histogram of durations lets you compute quantiles such as the 95th-percentile latency. Traces follow one request across services and show where the time went, split into spans.

Datadog, founded in 2010 in New York by two Centrale graduates, sells time: install its agent and dashboards, monitors and traces appear. Elasticsearch stores logs in an inverted index (for each word, the list of lines containing it), so a search becomes an intersection of lists. Prometheus works the other way round from most tools: it scrapes each service's /metrics page at a fixed interval and keeps labelled time series, queried in PromQL; Grafana displays them next to Loki logs. OpenTelemetry, a Cloud Native Computing Foundation project, standardises how the three signals are produced and shipped (the OTLP protocol, through a Collector), so you instrument once and can move from a local stack to a paid service without touching the code. The recommended course setup is the Grafana stack under Docker Compose.

A model adds its own indicators to latency, error rate and volume: input drift measured by the PSI per feature without waiting for labels, the distribution of output scores, real quality (AUC, calibration) once the truth arrives, sometimes months later, and, for language-model calls, the input and output tokens per request turned into a daily cost. The deck's thresholds are examples, such as a PSI above 0.25 from credit-scoring practice, and every alert needs an owner and an action. Instrumenting a FastAPI service takes a counter, a histogram, a /metrics route and one JSON log line per prediction carrying the model version.

The part ends by turning the tables: a model that observes an application. Using Loghub (LogPAI lab, Chinese University of Hong Kong), whose HDFS_v1 set holds 11,175,629 log lines over 38.7 hours according to the repository README read on 12 October 2026, students parse lines into templates (the Drain algorithm), then compare a rule on error words, an isolation forest on template counts, and a sequence model or language model. A language model then summarises each flagged incident for the on-call engineer, who still decides; the summaries are judged against about thirty written by hand with a fixed rubric. Loghub is free for research and teaching only, with the paper cited.

Back to contents.

Key takeaways

1. Scope before you code: six one-sentence lines (decision, target, metric, guardrails, data, plan), versioned in the repository and changed by pull request.
2. One simple tool per layer is enough for a project; Spark, Kafka and Kubernetes are worth understanding, not installing for three months of work.
3. Airflow's value is memory (run history, retries, backfills), and the same DAG can orchestrate language-model calls with an explicit human step.
4. Compare a naive, a best and a heavy model on the same split and metric, and climb a step only when the gain exceeds the noise.
5. Deploy as software: a training script, an API with an input contract, a separate interface, tests in CI, a container, pinned versions, secrets outside Git.
6. Observe with logs, metrics and traces; for a model, add drift, output distribution, delayed quality and the cost of every language-model call.
7. Read licences and terms before you build: several popular tools and datasets are source-available or non-commercial, not free software.

Glossary

Scope document

A one-page, six-line framing of a project (decision, target, metric, guardrails, data, plan), kept in the repository.

DAG

Directed acyclic graph: tasks and their dependencies, the unit Airflow schedules and runs.

Backfill

Replaying the runs of past dates of a pipeline, for instance after fixing a bug, without rerunning everything else.

MCP

Model Context Protocol, an open protocol through which a server exposes tools and resources to a language-model client.

RAG

Retrieval-augmented generation: fetch the most relevant passages and give them to a language model, whose weights do not change.

Baseline

The naive model (mean, last value, current rule) that every other model must beat to earn its place.

Continuous integration (CI)

Automatically reinstalling, linting, testing and building the project on a clean machine at every push.

Container

An image holding an application and all its dependencies, which runs identically on a laptop, in CI and at the host.

Observability

Being able to answer, without redeploying, questions about a running system from its logs, metrics and traces.

Drift

A change in the inputs or in their relation to the target after deployment, often measured with the PSI.

The other notes of the course

One public summary per deck, in the order of the course.

Previous: Open data ML projects by sectorNext: Data engineering, SRE and cloud

1. IntroductionAI winters, jobs and method
2. ToolsKaggle, GitHub, Python: the tools
3. Session 1Project framing and least squares
4. Session 2Maximum likelihood and logistic regression
5. Session 3Gradient descent, ridge and lasso
6. Session 4Decision threshold and reproducible ML
7. HomeworkA defensible baseline model
8. Session 5Kernels, SVM and decision trees
9. Session 6Forests, boosting, honest comparison
10. Session 7SHAP, fairness, PCA and clustering
11. HomeworkA documented champion model
12. Session 8Neural networks, drift, conformal prediction
13. HomeworkA delivered machine learning project
14. Session 9Defending a machine learning project
15. ReferenceCredit default scoring project
16. ReferenceAsset returns: XGBoost and sentiment
17. ReferencePython code for every ML model
18. ReferenceDeep learning essentials
19. ReferenceOpen data ML projects by sector
20. ReferenceData and AI engineer handbook
21. ReferenceData engineering, SRE and cloud
22. ReferencePanorama of machine learning models

All the notes, as a reading list

Attention and transformers, step by step

About these notes

Written by the lecturer from the slides of the deck, as a public summary: the key ideas, the formulas and one worked example. The exercises, their solutions, the tests and the grading stay in the deck, for the students of the course.

Revision: 16 October 2026.