Data engineering, SRE and cloud: the commands, the principles and the jobs, from junior to expert

The pipeline, and why Parquet

Data comes from sources, lands untouched in a raw zone, is cleaned and modelled into tables, then served to a dashboard, a model or an API.

The deck follows the lifecycle of Reis and Housley (2022): generate, ingest, store, transform, serve. The raw zone keeps what was received exactly as it arrived and is never modified, because it lets you rebuild everything after a bug. Cloud warehouses changed the order of the steps: load first, then transform in SQL inside the warehouse (ELT), the model dbt is built on.

CSV and JSON are text: universal, but large, slow to parse and untyped. Parquet stores data column by column, compressed, with types and per-block minimum and maximum. Avro stores rows with an evolving schema, for messages; Arrow is a columnar layout in memory shared by pandas, Polars, DuckDB and Spark. Computing a default rate needs one column out of 25, so Parquet reads about a twenty-fifth of the file, and block statistics let a filter skip whole blocks.

Keep the order of magnitude, not the digits: about three times smaller and, on this machine, about a hundred times faster, for the same answer. Millisecond timings are noisy and vary between machines.

Back to contents.

SQL on DuckDB, and the star schema

About a dozen SQL clauses cover most internship queries, and a star schema with a clearly stated grain keeps totals from being counted twice.

The deck runs twelve clauses on the Parquet file with DuckDB, an analytical database inside a Python package that needs no server: selection and filters, sorting, GROUP BY and HAVING, CASE WHEN, inner and left joins, window functions, CTEs, subqueries, and an index with EXPLAIN on SQLite.

The classic interview question: WHERE filters rows before grouping, HAVING filters groups after. Always print the count next to a rate, since a rate on a dozen people says nothing. A window function gives each row a value computed over its group without collapsing the table (a rank, a group mean, the previous row with LAG); after joins, it is the most frequent SQL interview topic.

For modelling, Kimball's star schema puts a fact table of measured, additive events in the centre and dimension tables (who, when, what) around it. The key sentence is the grain, here one row per client and statement month. When a dimension changes, type 1 overwrites and loses history, type 2 adds a dated row so the past can be replayed, type 3 keeps one previous value. Finance needs type 2: explaining yesterday's default with today's value is data leakage, as in the logistic regression note.

Back to contents.

Orchestration, idempotence and data tests

A good pipeline can be rerun for any day without doubling anything, and it stops before publishing when a test on the data fails.

An orchestrator (Airflow, Prefect, Dagster) runs tasks along a directed acyclic graph. Following Maxime Beauchemin (2018), each run handles one dated interval and writes one dated partition, so a retry reruns the failed day and a backfill reruns a range of days with unchanged code. This only works if the job is idempotent.

The dated partition, as measured in the deck

1. One folder per day. The load writes clients over 60 into a dt= folder named after the date, a Hive convention that engines read back as a column.
2. Delete, then write. The day's folder is removed first, so a second run replaces the partition instead of appending to it.
3. Replay and count. With the 12th of October loaded twice, each day still holds 272 rows, not 544. An append would have doubled the day silently.

Data tests. A test is a query that counts anomalies and must return zero; otherwise the pipeline stops before publishing, since a late dashboard beats a wrong one. The deck writes dbt's four generic tests by hand (not null, unique, accepted values, plausible range). Three pass; the accepted-values test finds 345 clients with education codes 0, 5 or 6 missing from the documentation. A failing test calls for a written decision, not necessarily a fix.

At senior level, a data contract, lineage and a catalogue answer the question of what breaks downstream when an upstream column changes.

Back to contents.

SRE: SLI, SLO and the error budget

Reliability is treated as a feature with a number: you measure it, agree on a target below one hundred percent, and spend the gap on change.

Site Reliability Engineering, born at Google and described in the 2016 book by Beyer and colleagues, rests on measuring reliability, automating repetitive work and learning from incidents without blame. The SLI is a ratio of good events to all events (say, scoring requests answered in under 300 ms). The SLO is the internal target on it, for example 99.9 % over 30 days. The SLA is a looser contract with penalties, so the team is alerted before paying. Aiming at 100 % is a mistake: each extra nine costs far more and users do not notice it.

Worked example, with the deck's illustrative values

1. The window. Thirty days hold $30 \times 24 \times 60 = 43\,200$ minutes.
2. The budget. At 99.9 %, a tenth of a percent of that is 43.2 minutes, or 2,000 failed requests out of 2 million. At 99 % it would be 432 minutes, at 99.99 % about 4.3.
3. The burn rate. Losing 10 minutes in one day gives $r = (10 / 43.2) / (1 / 30) \approx 6.94$, almost seven times the steady pace.
4. Exhaustion. At that pace the month's budget is gone after $30 / 6.94 \approx 4.3$ days.

The budget decides: while some remains, the team ships; once spent, features wait and the work goes to reliability. Alerts watch the burn rate rather than every error. Toil, the manual work that grows with traffic, is capped and automated away; on-call needs a written procedure per alert.

Incidents. Mitigate first (roll back, switch a feature off, add capacity), understand later, then write a blameless post-mortem with dated actions and owners. In the deck's fictitious case, a release missing one configuration variable is caught by the burn-rate alert and rolled back, at a cost of 39 of the month's 43.2 minutes.

Back to contents.

Infrastructure as code, Docker and Kubernetes

Servers, permissions and deployments are described in versioned files and rebuilt from images, never repaired by hand in a web console.

• Everything is code (Kief Morris, 2016): infrastructure lives in reviewed files and can be rebuilt identically.
• Immutable: a server is replaced by a new tested image, never patched by hand; rolling back means redeploying the previous one.
• Small blast radius: a change reaches a small share of traffic first (a canary) and is rolled back automatically if the SLI drops.
• Least privilege (Saltzer and Schroeder, 1975): each person and program gets only the rights it needs.
• A tested backup: only a restore drill proves it works, as the GitLab outage of 31 January 2017 showed.

The deck's Terraform file declares a versioned, tagged storage bucket for raw data in the Paris region. The cycle is init, plan, then apply after review; reading the plan is what catches an unexpected destroy. Terraform was not installed on the demonstration machine, so nothing was executed.

A container is an isolated process carrying its system, Python, packages and code; it shares the host kernel, so it starts in a second. Good Dockerfile habits: a slim official base, dependencies before code for the layer cache, a non-root user, configuration through environment variables. Kubernetes keeps the actual state of a cluster equal to the declared one: pods, a Deployment that recreates dead copies, a Service with a stable address, an Ingress for outside traffic. A managed container service is often enough for a small team.

Back to contents.

Cloud and FinOps: what a month really costs

In the cloud every resource has a meter that keeps running, and the bill often depends more on data leaving the provider than on storage.

AWS, Azure and Google Cloud dominate; OVHcloud, Scaleway and Hetzner matter for price and sovereignty, since data held by a US company can be reached through the CLOUD Act (2018). The building blocks are the same under different names: compute, object storage, managed databases, queues, functions, a warehouse, identity. Four meters run all the time: compute per hour, storage per gigabyte and month, egress per gigabyte leaving the provider, requests per thousand.

At AWS, egress costs as much as the machine: 400 billable gigabytes at 0.090 US dollars. The totals are in two currencies and the machines are comparable, not identical, so read an order of magnitude. For finance, compliance also shapes the architecture: the GDPR, the DORA regulation (applicable since 17 January 2025), the AI Act, which treats credit scoring of individuals as high-risk, and NIS2.

Back to contents.

The jobs, the salaries, and junior, senior, expert

The levels differ less by tools than by scope: a junior completes a task reliably, a senior answers for a system, an expert shapes a platform.

The data engineer builds the pipes; the analytics engineer turns raw tables into tested business tables; the foundation is the cloud or platform engineer's work; the SRE or DevOps engineer answers for availability; MLOps crosses them all.

No public statistic gives the salary of a data engineer in Luxembourg, and Apec has no sheet for SRE or MLOps: these markers compare jobs, they do not set a first offer. See also the AI history and jobs note.

Junior. Nobody expects a junior to know everything, only to become useful and trustworthy: SQL, Python, Git, the terminal, asking after a reasonable search, small changes, never touching production without a safety net. The usual mistakes: staying stuck alone, giant pull requests, a destructive command on the wrong account, unchecked figures, writing nothing down.

Senior. The step is about scope, not years: a vague problem and a system to answer for. The senior writes a design document, records decisions in an ADR (Michael Nygard, 2011), estimates in ranges, reviews code and mentors. Expert. Staff and principal engineers work in trade-offs: cost against latency, consistency against availability (CAP, PACELC), build against buy, choosing the simplest option that meets the SLO and the budget. What does not change: SQL, the shell, network protocols, idempotence, measuring before believing, writing clearly.

Back to contents.

Key takeaways

1. Keep raw data untouched and make every load idempotent: a dated partition that is replaced, never appended to, makes reruns harmless.
2. Columnar formats such as Parquet read only the columns a query needs; on the course data the file was about three times smaller and far faster to scan.
3. A data test is a query that must return zero rows, and a failing test stops publication.
4. An SLO of 99.9 % over 30 days allows 43.2 minutes of downtime, and alerts should watch how fast that budget burns.
5. Infrastructure belongs in versioned code, servers are replaced from images, and backups are proven by restores.
6. In the cloud, egress and machines left running often weigh more than storage.
7. Junior, senior and expert differ by scope, on fundamentals that stay the same.

Glossary

ELT

Extract, load, then transform inside the warehouse in SQL.

Columnar format

A file layout storing each column contiguously (Parquet), so a query reads only what it needs.

Star schema

A fact table of measured events surrounded by dimension tables that describe their context.

Slowly changing dimension (SCD)

A dimension whose values change over time; type 2 keeps every version with validity dates.

Directed acyclic graph (DAG)

Tasks linked by dependency arrows with no loop, executed by an orchestrator.

Idempotence

A job leaves the same state whether it runs once or several times.

Backfill

Rerunning a pipeline over a range of past dates with the current code.

Data contract

A producer's versioned commitment on columns, types and freshness.

Error budget

The share of failure an SLO allows over its window.

Burn rate

How fast the error budget is consumed compared with a steady pace.

Toil

Manual, repetitive, automatable operational work that grows with the service.

Blast radius

The share of users or systems affected when a change goes wrong.

Egress

Network traffic leaving a cloud provider, billed per gigabyte.

The other notes of the course

One public summary per deck, in the order of the course.

Previous: Data and AI engineer handbookNext: Panorama of machine learning models

1. IntroductionAI winters, jobs and method
2. ToolsKaggle, GitHub, Python: the tools
3. Session 1Project framing and least squares
4. Session 2Maximum likelihood and logistic regression
5. Session 3Gradient descent, ridge and lasso
6. Session 4Decision threshold and reproducible ML
7. HomeworkA defensible baseline model
8. Session 5Kernels, SVM and decision trees
9. Session 6Forests, boosting, honest comparison
10. Session 7SHAP, fairness, PCA and clustering
11. HomeworkA documented champion model
12. Session 8Neural networks, drift, conformal prediction
13. HomeworkA delivered machine learning project
14. Session 9Defending a machine learning project
15. ReferenceCredit default scoring project
16. ReferenceAsset returns: XGBoost and sentiment
17. ReferencePython code for every ML model
18. ReferenceDeep learning essentials
19. ReferenceOpen data ML projects by sector
20. ReferenceData and AI engineer handbook
21. ReferenceData engineering, SRE and cloud
22. ReferencePanorama of machine learning models

All the notes, as a reading list

Attention and transformers, step by step

About these notes

Written by the lecturer from the slides of the deck, as a public summary: the key ideas, the formulas and one worked example. The exercises, their solutions, the tests and the grading stay in the deck, for the students of the course.

Revision: 16 October 2026.