Writes and runs the jobs that move data from source systems into the warehouse on a schedule, and finds out why a load stopped arriving.
Build your own roadmap from your CVEvery pipeline ends in a table someone queries, and most of them start there too.
The orchestration layer, the transformations and the tests are all written in Python here.
A pipeline is only as useful as the tables it lands in, and you will be asked to shape them.
Extract, transform, load is the unit of daily work; you need the pattern before day one, not the tools.
Every pipeline change ships through a pull request; you review and get reviewed.
The application databases you read from are Postgres; its dialect and EXPLAIN output are daily tools.
Schedules and retries live in a DAG; you will write your first one in week one.
You load into one; knowing how it differs from an app database keeps loads fast and cheap.
Every job runs in a container; you must build one, run it and read its logs.
Pipelines run on Linux boxes; the shell is how you find out why a job died overnight.
A pipeline that runs green on bad data is worse than one that fails; you write the checks.
Facts and dimensions are how analysts read what you land; a wrong grain costs weeks.
Small transformations and ad-hoc checks happen in pandas before they graduate to the pipeline.